:W&Bá¾èЄ6M wueegkhmª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0IRÕú‚" zxpm21x9l207 wueegkhmŠÖ wueegkhmjuju11-uc-san-diegoSPAtop"™ _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.1627345651864043 epochs‚10  hidden_dim‚128 ' learning_rate‚0.0007922421082524079  num_attention_heads‚8  num_branch_encoder_layers‚5  num_embedding_layers‚14  num_encoder_layers‚4Bjjtttt_blocks_wueegkhmbs2s2ev0zj*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pŠ ÛÑèÔÜØæš3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64CÙL€7 wueegkhm&  _runtime‚0  _wandb.runtime‚0ðvf¨jÒä -Linux-6.8.0-138-generic-x86_64-with-glibc2.36CPython 3.12.10 ÚÑèÔÀ—Æä*-of*=/data/spatop/options_files/sweep_tttt_jj/config_wueegkhm.json*-l* /scratch/logs*-n*jjtttt_blocks_wueegkhm2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pr/opt/conda/bin/pythonº h1qqwmbs8bp55y7eqm00mvebka0vw82d˜ÿ£ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€ ÓßÏ$ª€ äÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-8cb16037-0025-0eda-9425-82f6843a99c3Ê13.2º h1qqwmbs8bp55y7eqm00mvebka0vw82dµª½Ç2  requirements.txtà KÌW wueegkhmZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64™Š—Y wueegkhmZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64£’—*  wueegkhmb trainer/global_step2½ê³Ä\  wueegkhmZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64TYª³/  wueegkhmb*"trainer/global_step2 NpK  wueegkhmj: ÜÑèÔÀæ¾Ó*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] %”MÁ“  wueegkhmj  ÜÑèÔÀîñÉî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 866 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 1.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 46.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ äáú\ wueegkhmjÊ ÜÑèÔ€ë„Ë·Trainable params: 48.1 M Non-trainable params: 86 Total params: 48.1 M Total estimated model params size (MB): 192.576 Modules in train mode: 5122 Modules in eval mode: 0 Total FLOPs: 0 ülŒq wueegkhmj` ÜÑèÔ˜©ÁËPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). 2ØbK{ wueegkhmjj ÜÑèÔ ©ÎËZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ²¹gy wueegkhmjh ÜÑèÔ ‘ÖËXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). C]•ö€ wueegkhmjo ÜÑèÔð‹ÝË_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). •Óõ¹z wueegkhmji ÜÑèÔ€ÈãËYwandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). >Řt wueegkhmjc ÜÑèÔÈ›êËSwandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). i‹µ`q wueegkhmj` ÜÑèÔ°ŽñËPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). 9CŸt wueegkhmjc ÜÑèÔð½öËSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). úÓcn wueegkhmj] ÜÑèÔøÒüËMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). Q…×ìm wueegkhmj\ ÜÑèÔèï‚ÌLwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). 1,rÜ wueegkhm*Ê  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  pairwise_feature_set‚""  pairwise_eq2to2‚false  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"øt"wë wueegkhmjÙ ÜÑèÔˆßôÕÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ®Q¼ŽN wueegkhmj¼ ÜÑèÔ°ÚÖ«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. P´ë wueegkhmjÙ àÑèÔ€»ÏäÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. R/vP wueegkhmj¾ àÑèÔÀìýä­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ¢ 3’‚(:ˆ ëÑèÔœ¤Ù² gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ØïfP‚(:Æ ëÑèÔ¢ÆÄÆ disk.dm-1.out‚ 0.36328125% disk.nvme0c0n1.out‚631.7392578125 disk.nvme0n1.in‚ 25.296875) proc.memory.availableMB‚ 2441.82421875 cpu‚18.010842758332725 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚85.09628772735596" proc.memory.rssMB‚ 2842.7109375) proc.memory.percent‚17.35053062438965 proc.cpu.threads‚17 network.sent‚40956 network.recv‚42244) disk./.usagePercent‚78.65804634342595 disk.dm-0.in‚ 25.296875 disk.dm-0.out‚631.7392578125 disk.sda3.in‚0 disk.sda3.out‚ 0.36328125 disk.nvme0c0n1.in‚ 25.296875% disk./.usageGB‚1172.0617332458496 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0 disk.sda.in‚0 disk.sda.out‚ 0.36328125 disk.sda1.out‚0# disk.nvme0n1.out‚631.7392578125¥ë’ ‚(:ˆ úÑèÔé‚Ü­ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚167.713' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)àèfY!‚(:Ï úÑèÔø«°) proc.memory.availableMB‚ 2441.27734375# proc.memory.rssMB‚ 2842.75390625 network.sent‚44690 disk.dm-1.out‚0.71875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 25.30078125$ disk.nvme0n1.out‚686.88134765625 proc.cpu.threads‚17 network.recv‚46285) disk./.usagePercent‚78.67331744885092 disk.dm-1.in‚0& disk.nvme0c0n1.out‚686.88134765625 disk.nvme0n1.in‚ 25.30078125! disk.dm-0.out‚686.88134765625$ memory_percent‚85.09962558746338) proc.memory.percent‚17.35079288482666 disk.sda3.in‚0 disk.drbd1260.out‚ 0.00390625 disk.sda.out‚0.71875 disk.sda1.out‚0 cpu‚18.96949197611766% disk./.usageGB‚1172.2892837524414 disk.sda3.out‚0.71875 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 25.30078125b—R’"‚(:ˆ ‰ÒèÔÝÎÕ¯ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.356' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆåÜ`#‚(:Ö ‰ÒèÔ„Œ»´) disk./.usagePercent‚78.67593435786593 disk.sda.out‚ 1.2890625 disk.dm-0.in‚ 25.30078125) proc.memory.availableMB‚ 2441.73828125) proc.memory.percent‚17.35084056854248 disk.sda3.out‚ 1.2890625 disk.drbd1260.out‚ 0.00390625 disk.sda.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2842.76171875 cpu‚19.706444755687542 proc.cpu.threads‚17 network.sent‚51342% disk./.usageGB‚1172.3282775878906 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 25.30078125$ disk.nvme0n1.out‚1173.1630859375$ memory_percent‚85.09681224822998 network.recv‚51453 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 1.2890625 disk.sda1.in‚0& disk.nvme0c0n1.out‚1173.1630859375 disk.nvme0n1.in‚ 25.30078125! disk.dm-0.out‚1173.16308593753c€Ï3$b/ lr-AdamW/pg1"trainer/global_step2:JP˜‰7&3%b/ lr-AdamW/pg2"trainer/global_step2:JPãH\ -&) _wandbruntime‚58  _runtime‚584ÎpõÀ'» % _timestamp‚1.7884879591235516e+09 _runtime‚ 58.764518665 _step‚0 & lr-AdamW/pg1‚3.210906807640032e-05 & lr-AdamW/pg2‚3.210906807640032e-05 trainer/global_step‚49ÿ7s‘Â(½ _runtime‚ 58.764518665 _step‚0 & lr-AdamW/pg1‚3.210906807640032e-05 & lr-AdamW/pg2‚3.210906807640032e-05 trainer/global_step‚49 % _timestamp‚1.7884879591235516e+09%t»p)bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP«ÆP9o*bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPr` Mj+bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JP3¡7¦k,bg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JP-õÖm-bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP!oŒ:k.bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPšË0Žp/bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPàSeõm0bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPÙˆðêg1bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP‘:)Tv2br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPfSϲf3bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPs÷P‘W4bS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPm˜T s5bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPk@bn6bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP/­»a7b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPúVGjj8bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPýP¦_9b[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JP¹DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP äùn=bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP*dÎýf>bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP]©žg?bc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPÿiþÞn@bj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP×ñosAbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPÉ«ÞnBbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPeA¤nCbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP‡ð%]sDbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP(à hEbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPÌÂöiFbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPÒÀ¶fGbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP’ºs½bHb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPðô¤^IbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPtâý_sJbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPdbKb^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPÓ3½fLbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPðåÕ´lMbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP^úþlNbh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPh|1kObg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPg'BlPbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPoL`Qb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP:'jõsRbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP¿VîQgSbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JP:þdzoTbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPœc}ÓqUbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPÒ2xEaVb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JP¢ó-hWbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPp¯öjXbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPh à%eYba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPSíXÕrZbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP‰/.g[bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP¯"hm\bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPÔ뇙d]b` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP½ãXÖq^bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPDE– g_bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JP>"–6o`bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPÀ\-mabi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP‚¤ù\ibbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP4Ÿëecba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPƒ‡ÞBidbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JPèä³]ebY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPmò!efba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPXó¥Xogbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP—< êihbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP3¬Þnibj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP‰$U(ijbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPšãØÚ^kbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPº æ?nlbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPæâ¨ðlmbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPÑíjädnb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPÆ -Niobe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPáäTîgpbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPÈø!Écqb_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPn4Ë kžbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP8:ArŸbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP\xŠot bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPî­Ñn¡bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP‚³\›l¢bg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JP|<ãÔk£bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPÁ½Ì¹.¤) _wandbruntime‚58  _runtime‚58àê"Ñ¥”O P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875269 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209589248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3406136627712 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚265363 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 53034820096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3456104641536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018906624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 $ _timestamp‚1.788487959125333e+09 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1956325 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚852772 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 50574482876 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 216156672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26615 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3321864071792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1082035 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 49986016768 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3456894522368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 _runtime‚ 58.765956676 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 49986016768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1103553 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚265884 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891236352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚754178 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233529344 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3838727085568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3321847032432 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3456894522368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627269 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 369652224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26346 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1082035 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 50757894656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚987 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚852772 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3406136627712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3406118624768 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚852770 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚754701 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 50757894656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39Bç÷dž'¦˜O ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 49986016768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1103553 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚265884 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891236352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚754178 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233529344 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3838727085568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3321847032432 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3456894522368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627269 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 369652224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26346 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1082035 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 50757894656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚987 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚852772 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3406136627712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3406118624768 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚852770 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚754701 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 50757894656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875269 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209589248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3406136627712 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚265363 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 53034820096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3456104641536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018906624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 $ _timestamp‚1.788487959125333e+09 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1956325 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚852772 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 50574482876 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 216156672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26615 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3321864071792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1082035 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 49986016768 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3456894522368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 _runtime‚ 58.765956676 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0_ø-]§bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JP×›f}m¨bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPdlJók©bf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPàK¹…oªbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP Øxf«ba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPÄ>Íej¬be BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPö4ëNo­bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPÛ Û:h®bc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JP‰«x±r¯bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP¶~ÈId°b_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPåLæ¨lÊbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP¯"ŽmËbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPaâ¹ajÌbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JP݉xjgÍbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JPšJì°hÎbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JP)¡äfgÏbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPî=àŠgÐbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPOÓV“iÑbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JP´“/jÒbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPÌ(j½fÓba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPc·šûjÔbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JPÍ‹bÕb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPýªÃÿpÖbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPx}_ƒk×bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP2 ¨jØbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP¬d_ÙbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP|XK=kÚbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPU¨ŽkÛbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JP ,áogÜbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP~{mÝbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP÷{A'kæbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JP$¨hçbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JP€dÖheèb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPC®PÑrébm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP J“fêba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP¹$ ¾mëbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPW9Úbìb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP÷Ëj_eíb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPZ€{Ydîb_ DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPõN½Œiýbd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JP ƒJiþbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPÛM¾fÿba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPX^“¸o€bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP„Ãf^bY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JP<#àÍe‚b` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPŽ$M@mƒbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP¹»gk„bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPTÃlön…bi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP%a¾ê_†bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPû\{n‡bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP“;oˆbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP\V]@o‰bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPÓKô÷jŠbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JP%Jϯf‹ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP€Q}nŒbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPlrŒhbc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JP@ñ$–qŽbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPd—§³wbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPèa´‘db_ yu’bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP0ëÃ]“bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPˆœ¨b”b] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPÿu"DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1994936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 51575622962 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1994936 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1016 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚869644 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1125290 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 51762653696 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚498463 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 50990762496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227224576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3387913468192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3473836567040 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 50990762496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚271107 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 54067072512 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3525599220736 UEDeviceStatsMonitor.on_train_batch_end/all¡[k§hocated_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3861274137600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 trainer/global_step‚49 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3439506130514 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3387930507552 _runtime‚ 59.673740622 NDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1125290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3438718450082 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 # _timestamp‚1.78848796003267e+09 _step‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚769572 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚769050 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1973390 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 369652224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3473818564096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚869646 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1103746 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3861250197504 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072tš§&¤¡M VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3473836567040 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1994936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 51575622962 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1994936 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1016 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚869644 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1125290 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 51762653696 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚498463 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 50990762496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227224576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3387913468192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3473836567040 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 50990762496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚271107 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 54067072512 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3525599220736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3861274137600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 trainer/global_step‚49 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3439506130514 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3387930507552 _runtime‚ 59.673740622 NDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1125290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3438718450082 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 # _timestamp‚1.78848796003267e+09 _step‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚769572 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚769050 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1973390 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 369652224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3473818564096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚869646 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1103746 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3861250197504#iØA¥b< loss/FRt2/assignment_loss"trainer/global_step2:JPyAb loss/SRqqt1/assignment_loss"trainer/global_step2:JP>Ë A¨b< loss/FBt4/assignment_loss"trainer/global_step2:JPnaNØB©b= loss/SRqqt4/detection_loss"trainer/global_step2:JPí~3Bªb= loss/SRqqt2/detection_loss"trainer/global_step2:JP++@«b; loss/FBt2/detection_loss"trainer/global_step2:JP‡£-¬b( epoch"trainer/global_step2:JP9+[C­b> loss/SRqqt3/assignment_loss"trainer/global_step2:JPf¿ÓA®b< loss/FBt2/assignment_loss"trainer/global_step2:JP‹×A¯b< loss/FBt3/assignment_loss"trainer/global_step2:JPƒn£@°b; loss/FRt3/detection_loss"trainer/global_step2:JPiy ¸B±b= loss/SRqqt1/detection_loss"trainer/global_step2:JP;{÷ @²b; loss/FBt4/detection_loss"trainer/global_step2:JPÒHù7³b2 loss/total_loss"trainer/global_step2:JPùœpžA´b< loss/FRt1/assignment_loss"trainer/global_step2:JPÏ„VÂAµb< loss/FRt3/assignment_loss"trainer/global_step2:JP9@¶b; loss/FRt2/detection_loss"trainer/global_step2:JP-½ Ñ@·b; loss/FRt4/detection_loss"trainer/global_step2:JP^ñV!B¸b= loss/SRqqt3/detection_loss"trainer/global_step2:JPàË@¹b; loss/FBt1/detection_loss"trainer/global_step2:JPë%Cºb> loss/SRqqt4/assignment_loss"trainer/global_step2:JP£Õ@»b; loss/FRt1/detection_loss"trainer/global_step2:JP¡[eÓC¼b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP×°™A½b< loss/FBt1/assignment_loss"trainer/global_step2:JPÒK)f@¾b; loss/FBt3/detection_loss"trainer/global_step2:JPlÿE.¿) _wandbruntime‚59  _runtime‚59Ð:ObEÀ¿ 0loss/FBt1/assignment_loss‚2.6149346310159434 .loss/FBt3/detection_loss‚4.215672318676584 .loss/FRt2/assignment_loss‚7.05616513542507 /loss/FRt4/assignment_loss‚7.416390584439648 1loss/SRqqt1/assignment_loss‚5.348137651171003 /loss/FBt4/assignment_loss‚2.576186492525298 0loss/SRqqt4/detection_loss‚2.616292189164408 0loss/SRqqt2/detection_loss‚6.689333238280737 -loss/FBt2/detection_loss‚5.31305182678625 $ _timestamp‚1.788487960035535e+09 _step‚3 epoch‚0 1loss/SRqqt3/assignment_loss‚5.525391557942266 /loss/FBt2/assignment_loss‚2.606078028678894 0loss/FBt3/assignment_loss‚2.5729624158457707 .loss/FRt3/detection_loss‚4.545990907832196 1loss/SRqqt1/detection_loss‚3.9149418742883775 /loss/FBt4/detection_loss‚5.6493398059031055 &loss/total_loss‚116.07866568159268 _runtime‚ 59.676955542 /loss/FRt1/assignment_loss‚7.222236607869466 /loss/FRt3/assignment_loss‚7.479595799195139 .loss/FRt2/detection_loss‚5.348269426288393 .loss/FRt4/detection_loss‚3.432316398742248 1loss/SRqqt3/detection_loss‚3.5846126960671465 .loss/FBt1/detection_loss‚7.379160855775294 1loss/SRqqt4/assignment_loss‚5.539674216303332 /loss/FRt1/detection_loss‚2.0234969074527425 trainer/global_step‚49 1loss/SRqqt2/assignment_loss‚5.408434115923368œÔ™IÁà 1loss/SRqqt4/assignment_loss‚5.539674216303332 /loss/FRt1/detection_loss‚2.0234969074527425 trainer/global_step‚49 1loss/SRqqt2/assignment_loss‚5.408434115923368 0loss/FBt1/assignment_loss‚2.6149346310159434 .loss/FBt3/detection_loss‚4.215672318676584 .loss/FRt2/assignment_loss‚7.05616513542507 /loss/FRt4/assignment_loss‚7.416390584439648 1loss/SRqqt1/assignment_loss‚5.348137651171003 /loss/FBt4/assignment_loss‚2.576186492525298 0loss/SRqqt4/detection_loss‚2.616292189164408 0loss/SRqqt2/detection_loss‚6.689333238280737 -loss/FBt2/detection_loss‚5.31305182678625 $ _timestamp‚1.788487960035535e+09 _step‚3 epoch‚0 1loss/SRqqt3/assignment_loss‚5.525391557942266 /loss/FBt2/assignment_loss‚2.606078028678894 0loss/FBt3/assignment_loss‚2.5729624158457707 .loss/FRt3/detection_loss‚4.545990907832196 1loss/SRqqt1/detection_loss‚3.9149418742883775 /loss/FBt4/detection_loss‚5.6493398059031055 &loss/total_loss‚116.07866568159268 _runtime‚ 59.676955542 /loss/FRt1/assignment_loss‚7.222236607869466 /loss/FRt3/assignment_loss‚7.479595799195139 .loss/FRt2/detection_loss‚5.348269426288393 .loss/FRt4/detection_loss‚3.432316398742248 1loss/SRqqt3/detection_loss‚3.5846126960671465 .loss/FBt1/detection_loss‚7.379160855775294â4DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6891353308160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 6791115591168 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚527702 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2170937 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3888927 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1696472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2192455 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7553318993920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1027 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚528234 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6892143188992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 6791115591168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚998 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 114.148308633 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7553342934016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 104822433792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6791133594112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6791133594112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1696470 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1696470 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 100645067984 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2170937 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3867407 VEDeviceStatsMonitor.on_train_batch_ss âÊ tart/active_bytes.small_pool.freed‚ 100237716992 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 99874427120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1499451 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 6725043247552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 6625185859792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 101009594880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _step‚5 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1696472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚971217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225140736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 201200640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 % _timestamp‚1.7884880145074923e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7658141427712 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 6725830927776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2192455 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 105023634432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 101009594880 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 100237716992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096é,ž­'ϧO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 105023634432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 101009594880 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 100237716992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6892143188992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6891353308160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 6791115591168 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚527702 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2170937 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3888927 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1696472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2192455 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7553318993920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1027 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚528234 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6892143188992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 6791115591168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚998 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 114.148308633 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7553342934016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 104822433792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6791133594112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6791133594112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1696470 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1696470 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 100645067984 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2170937 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3867407 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 100237716992 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 99874427120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1499451 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 6725043247552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 6625185859792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 101009594880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _step‚5 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1696472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚971217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225140736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 201200640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 % _timestamp‚1.7884880145074923e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7658141427712 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 6725830927776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2192455 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 Q¦+0Ð+ _wandbruntime‚115  _runtime‚115_¬´&Ñ®M H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 102014353920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 6691235256192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1713344 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚533023 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6858833533440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1027 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2192648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7734515646464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 216156672 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884880157374792e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6858815530496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 6691252295552 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 6960847887360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3905992 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚998 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6858833533440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6858815530496 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3927538 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3905992 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 trainer/global_step‚99 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2214192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1513886 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7734749162496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1713346 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1713344 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2214192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 100875566998 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 _runtime‚ 115.379783819 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3927538 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚533565 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 101646208070 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 106057885696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209575936 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7628691276800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 6960847887360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1713344 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚533023 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6858833533440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1027 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2192648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7734515646464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 216156672 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884880157374792e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6858815530496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 6691252295552 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 6960847887360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3905992 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚998 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6858833533440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6858815530496 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3927538 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3905992 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 trainer/global_step‚99 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2214192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1513886 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7734749162496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1713346 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1713344 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2214192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 100875566998 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 _runtime‚ 115.379783819 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3927538 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚533565 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 101646208070 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 106057885696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209575936 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7628691276800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 6960847887360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3281357 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚791509 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11320736133120 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚10078435620660 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2245680 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 149945012228 trainer/global_step‚149 _runtime‚ 168.493269303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 _step‚9 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10327391855616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 150489417216 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚792063 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11478015503872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2245124 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229335040 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 150715653092 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10326601974784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2540170 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5800009 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3259839 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5821529 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1003 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 157255430656 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 157050035712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3281357 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3259839 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 % _timestamp‚1.7884880688526762e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1453617 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10176112557568 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚10327391855616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10326601974784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11477786168832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 151261295104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11320760073216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 216156672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2540172 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚554 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205394944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚10176112557568 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2540170 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9928490608432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376ç¢À'ãºO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 150489417216 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚792063 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11478015503872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2245124 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229335040 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 150715653092 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10326601974784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2540170 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5800009 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3259839 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5821529 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1003 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 157255430656 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 157050035712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3281357 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3259839 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 % _timestamp‚1.7884880688526762e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1453617 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10176112557568 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚10327391855616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10326601974784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11477786168832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 151261295104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11320760073216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 216156672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2540172 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚554 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205394944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚10176112557568 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2540170 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9928490608432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚10176130560512 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3281357 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚791509 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11320736133120 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚10078435620660 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2245680 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 149945012228 trainer/global_step‚149 _runtime‚ 168.493269303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 _step‚9 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10327391855616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 Òí'q0ä+ _wandbruntime‚169  _runtime‚169ÊœsXïåÁM R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚10145503196298 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚797263 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 213770240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚10146290876730 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2557046 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚10243830499840 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 152266054144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚11554402434560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11396084475904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 151716793178 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5860140 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3281550 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚10395306659840 QîÊæaØEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2557046 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2259985 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 152266054144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3303094 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 151494162944 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 158080248320 _runtime‚ 169.471033483 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2557044 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2557044 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚10243812496896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5860140 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚796722 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 151494162944 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 237710336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9994574083552 % _timestamp‚1.7884880698295941e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 237710336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9994574083552 % _timestamp‚1.7884880698295941e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚797263 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 213770240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚10146290876730 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2557046 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚10243830499840 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 152266054144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚11554402434560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11396084475904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 151716793178 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5860140 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3281550 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚10395306659840 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2557046 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2259985 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 152266054144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3303094 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 151494162944 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 158080248320 _runtime‚ 169.471033483 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2557044 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2557044 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚10243812496896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5860140 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚796722 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 }ð¬Ê0ç+ _wandbruntime‚169  _runtime‚169C£Ý{RèÌ /loss/FRt3/detection_loss‚3.2424613663128445 /loss/FRt4/detection_loss‚2.9345754930483445 .loss/FBt3/detection_loss‚2.588994809279315 /loss/FRt1/assignment_loss‚7.087304018311581 1loss/SRqqt3/detection_loss‚3.9290308505296707 1loss/FBt2/assignment_loss‚0.25118054510094223 %loss/total_loss‚99.97630711241919 epoch‚0 % _timestamp‚1.7884880698330512e+09 /loss/FRt4/assignment_loss‚7.387151122093201 1loss/FBt1/assignment_loss‚0.21064374545988226 1loss/SRqqt4/assignment_loss‚3.901636584385021 .loss/FRt2/detection_loss‚5.678088853756587 1loss/SRqqt1/detection_loss‚2.9697431309521196 0loss/FBt3/assignment_loss‚0.7044055553029894 1loss/SRqqt4/detection_loss‚1.4425825192750357 /loss/FBt1/detection_loss‚11.814408085175923 .loss/FBt2/detection_loss‚6.571911565065384 .loss/FBt4/detection_loss‚8.187435546517372 _step‚11 /loss/FRt2/assignment_loss‚6.937176545461019 0loss/FRt3/assignment_loss‚7.3529788231363105 1loss/SRqqt2/assignment_loss‚3.466889741572928 2loss/SRqqt3/assignment_loss‚3.7784721624283564 _runtime‚ 169.473250552 /loss/FBt4/assignment_loss‚0.400035276543349 0loss/SRqqt2/detection_loss‚3.522217610414992 trainer/global_step‚149 1loss/SRqqt1/assignment_loss‚3.212800717353821 /loss/FRt1/detection_loss‚2.4041824449421996s‚—VéÐ _runtime‚ 169.473250552 /loss/FBt4/assignment_loss‚0.400035276543349 0loss/SRqqt2/detection_loss‚3.522217610414992 trainer/global_step‚149 1loss/SRqqt1/assignment_loss‚3.212800717353821 /loss/FRt1/detection_loss‚2.4041824449421996 /loss/FRt3/detection_loss‚3.2424613663128445 /loss/FRt4/detection_loss‚2.9345754930483445 .loss/FBt3/detection_loss‚2.588994809279315 /loss/FRt1/assignment_loss‚7.087304018311581 1loss/SRqqt3/detection_loss‚3.9290308505296707 1loss/FBt2/assignment_loss‚0.25118054510094223 %loss/total_loss‚99.97630711241919 epoch‚0 % _timestamp‚1.7884880698330512e+09 /loss/FRt4/assignment_loss‚7.387151122093201 1loss/FBt1/assignment_loss‚0.21064374545988226 1loss/SRqqt4/assignment_loss‚3.901636584385021 .loss/FRt2/detection_loss‚5.678088853756587 1loss/SRqqt1/detection_loss‚2.9697431309521196 0loss/FBt3/assignment_loss‚0.7044055553029894 1loss/SRqqt4/detection_loss‚1.4425825192750357 /loss/FBt1/detection_loss‚11.814408085175923 .loss/FBt2/detection_loss‚6.571911565065384 .loss/FBt4/detection_loss‚8.187435546517372 _step‚11 /loss/FRt2/assignment_loss‚6.937176545461019 0loss/FRt3/assignment_loss‚7.3529788231363105 1loss/SRqqt2/assignment_loss‚3.466889741572928 2loss/SRqqt3/assignment_loss‚3.7784721624283564 e’{Úžê‚(:“ ÓèÔÕ¤ä° gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚124.912' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.982400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×dë…_ë‚(:Ô ÓèÔ䪪±% disk./.usageGB‚1172.8285369873047 disk.dm-1.out‚ 5.45703125 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.out‚ 4463.2578125$ memory_percent‚85.11960506439209) proc.memory.availableMB‚ 2438.00390625 disk.sda3.in‚0 disk.sda3.out‚ 5.45703125 disk.drbd1260.out‚ 0.0234375 disk.sda2.out‚0 cpu‚22.512633684164015 proc.cpu.threads‚17 network.recv‚122074( disk./.usagePercent‚78.7095071859062 disk.dm-1.in‚0 disk.sda.out‚ 5.45703125 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 303.50390625# disk.nvme0c0n1.out‚ 4463.2578125 disk.nvme0n1.in‚ 303.50390625 disk.drbd1260.in‚0 disk.sda1.out‚0! disk.nvme0n1.out‚ 4463.2578125 disk.dm-0.in‚ 303.50390625# proc.memory.rssMB‚ 2842.81640625* proc.memory.percent‚17.351174354553223 network.sent‚212747}iS“ì‚(:ˆ ŸÓèÔñ¬ÿ­ gpu.0.gpu‚43.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.156' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.4312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0` Kí‚(:À ŸÓèÔ­Ïɱ disk.sda1.in‚0 disk.sda3.out‚ 5.79296875& proc.memory.availableMB‚ 2437.90625 cpu‚22.653595462714133 proc.cpu.threads‚17) disk./.usagePercent‚78.70793094751788% disk./.usageGB‚1172.8050498962402 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0234375 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 427.796875 disk.nvme0c0n1.out‚4919.125 network.recv‚127046 disk.sda.out‚ 5.79296875 disk.sda2.in‚0 disk.nvme0n1.in‚ 427.796875 disk.dm-0.out‚4919.125 disk.sda3.in‚0 network.sent‚262027 disk.dm-1.in‚0 disk.sda1.out‚0 disk.nvme0n1.out‚4919.125 disk.dm-0.in‚ 427.796875$ memory_percent‚85.12020111083984# proc.memory.rssMB‚ 2842.81640625* proc.memory.percent‚17.351174354553223 disk.dm-1.out‚ 5.79296875 disk.sda.in‚0­+¶2’î‚(:‡ ®ÓèÔÒí·® gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.8026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E9X iï‚(:Þ ®ÓèÔ¶Ÿâ¯ disk.dm-0.in‚ 427.8046875$ memory_percent‚85.12146472930908# proc.memory.rssMB‚ 2842.81640625* proc.memory.percent‚17.351174354553223 network.recv‚130772 disk.sda3.in‚0 disk.sda.out‚ 6.39453125 disk.sda2.out‚0' disk.nvme0c0n1.out‚5286.30419921875" disk.dm-0.out‚5286.30419921875) proc.memory.availableMB‚ 2437.69921875 cpu‚22.780326338096902 network.sent‚267501 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17( disk./.usagePercent‚78.6597470014759% disk./.usageGB‚1172.0870742797852 disk.sda3.out‚ 6.39453125 disk.dm-1.out‚ 6.39453125 disk.drbd1260.out‚ 0.02734375 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 427.8046875 disk.nvme0n1.in‚ 427.8046875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0% disk.nvme0n1.out‚5286.30419921875ücÊù0ð+ _wandbruntime‚219  _runtime‚219ò¯ÆñÀ trainer/global_step‚199 % _timestamp‚1.7884881203445141e+09 _runtime‚ 219.984933474 _step‚12 ' lr-AdamW/pg1‚0.00013040213361640128 ' lr-AdamW/pg2‚0.00013040213361640128`b^VÊòÄ % _timestamp‚1.7884881203445141e+09 _runtime‚ 219.984933474 _step‚12 ' lr-AdamW/pg1‚0.00013040213361640128 ' lr-AdamW/pg2‚0.00013040213361640128 trainer/global_step‚199 S~AB0ó+ _wandbruntime‚219  _runtime‚219ÓÇÌzO%ô»O `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15088153272320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7732611 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3383870 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13561127526912 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4370259 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13761850641408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4370259 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 200741117440 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1036 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13761850641408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15088177212416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚13762640522240 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 209108836864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4348741 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1936015 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212557824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 200786238200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13561109523968 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4348741 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15297474667008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13561127526912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7754131 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1007 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 _step‚13 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13432615673992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13761850641408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15088177212416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚13762640522240 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 209108836864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4348741 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1936015 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 212557824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 200786238200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13561109523968 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4348741 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15297474667008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13561127526912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7754131 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1007 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 _step‚13 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13432615673992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7732611 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3383870 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13561127526912 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4370259 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13761850641408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4370259 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 “e¶v0ö+ _wandbruntime‚220  _runtime‚220þºpŠË&÷ÅM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚13831345220608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13628827466240 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13628809463296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1060852 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 216156672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 _step‚14 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4370452 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3400744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4391996 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 201745863168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7771196 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4391996 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15373879396864 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 210167334400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13628809463296 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 202517754368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1036 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15163525555200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7792742 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚13831345220608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1945663 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13498895569406 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3400744 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 202517754368 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1061375 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13297895871552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 201745863168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1945665 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 201787378286 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7771196 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚13830555326464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13297878832192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3006515 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13499683249838 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7792742 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15373668949504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 201016737214 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7771196 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚13830555326464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13297878832192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3006515 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13499683249838 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7792742 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15373668949504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 201016737214 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15373879396864 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 210167334400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13628809463296 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 202517754368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1036 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15163525555200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7792742 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚13831345220608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1945663 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13498895569406 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3400744C‚N0ù+ _wandbruntime‚220  _runtime‚220àiylSúÍ .loss/FBt4/detection_loss‚3.713905771955466 /loss/FBt4/assignment_loss‚0.370006766429959 1loss/SRqqt1/detection_loss‚2.1198935539207677 .loss/FBt2/detection_loss‚27.65478907028834 0loss/FRt4/assignment_loss‚7.4098099042784495 .loss/FRt1/detection_loss‚2.646671390770511 epoch‚0 /loss/FRt1/assignment_loss‚7.163858803835782 /loss/FRt2/assignment_loss‚7.183643466547916 1loss/FBt1/assignment_loss‚0.42521534718495485 -loss/FBt1/detection_loss‚10.7765076816082 % _timestamp‚1.7884881213009849e+09 .loss/FRt2/detection_loss‚4.787105635592812 /loss/FRt3/detection_loss‚3.8120343908667564 0loss/SRqqt2/detection_loss‚4.217574171530894 &loss/total_loss‚112.39915813457773 trainer/global_step‚199 1loss/SRqqt2/assignment_loss‚3.473156696710831 1loss/FBt2/assignment_loss‚0.37690927740186453 .loss/FRt4/detection_loss‚3.116309666689837 _runtime‚ 220.941309153 0loss/FRt3/assignment_loss‚7.2912197680700395 2loss/SRqqt1/assignment_loss‚3.2809987754532783 2loss/SRqqt3/assignment_loss‚3.3794388156948667 2loss/SRqqt4/assignment_loss‚3.6287502162861376 0loss/FBt3/assignment_loss‚0.7743158553356706 _step‚15 1loss/SRqqt3/detection_loss‚2.2885725116639426 1loss/SRqqt4/detection_loss‚1.1112240079017772 /loss/FBt3/detection_loss‚1.3972465885586725¼ï©DûÑ 0loss/FRt4/assignment_loss‚7.4098099042784495 .loss/FRt1/detection_loss‚2.646671390770511 epoch‚0 /loss/FRt1/assignment_loss‚7.163858803835782 /loss/FRt2/assignment_loss‚7.183643466547916 1loss/FBt1/assignment_loss‚0.42521534718495485 -loss/FBt1/detection_loss‚10.7765076816082 % _timestamp‚1.7884881213009849e+09 .loss/FRt2/detection_loss‚4.787105635592812 /loss/FRt3/detection_loss‚3.8120343908667564 0loss/SRqqt2/detection_loss‚4.217574171530894 &loss/total_loss‚112.39915813457773 trainer/global_step‚199 1loss/SRqI`æBqt2/assignment_loss‚3.473156696710831 1loss/FBt2/assignment_loss‚0.37690927740186453 .loss/FRt4/detection_loss‚3.116309666689837 _runtime‚ 220.941309153 0loss/FRt3/assignment_loss‚7.2912197680700395 2loss/SRqqt1/assignment_loss‚3.2809987754532783 2loss/SRqqt3/assignment_loss‚3.3794388156948667 2loss/SRqqt4/assignment_loss‚3.6287502162861376 0loss/FBt3/assignment_loss‚0.7743158553356706 _step‚15 1loss/SRqqt3/detection_loss‚2.2885725116639426 1loss/SRqqt4/detection_loss‚1.1112240079017772 /loss/FBt3/detection_loss‚1.3972465885586725 .loss/FBt4/detection_loss‚3.713905771955466 /loss/FBt4/assignment_loss‚0.370006766429959 1loss/SRqqt1/detection_loss‚2.1198935539207677 .loss/FBt2/detection_loss‚27.65478907028834º6VÓ“ü‚(:ˆ ½ÓèÔ¡¬Â¯ gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.927' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.9854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Îë#Dgý‚(:Ü ½ÓèÔþ¢î± disk.sda.out‚ 6.89453125! disk.nvme0c0n1.in‚ 427.8046875% disk.nvme0n1.out‚5569.77685546875 disk.dm-1.in‚0 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚5569.77685546875 disk.nvme0n1.in‚ 427.8046875 disk.dm-0.in‚ 427.8046875( proc.memory.availableMB‚ 2437.8671875* proc.memory.percent‚17.351198196411133 network.sent‚287291$ disk./.usageGB‚1172.086841583252" disk.dm-0.out‚5569.77685546875 disk.sda3.in‚0 disk.sda3.out‚ 6.89453125$ memory_percent‚85.12043952941895" proc.memory.rssMB‚ 2842.8203125 cpu‚22.894812903244265 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 6.89453125 proc.cpu.threads‚17 network.recv‚135980) disk./.usagePercent‚78.65973138501633 disk.drbd1260.out‚ 0.02734375 disk.sda.in‚0ƒWmžþ‚(:“ ÌÓèÔÝù—® gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.187' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.437400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç_ë+Vÿ‚(:Ë ÌÓèÔ®©Í¯ network.sent‚324178) disk./.usagePercent‚78.65815924274855$ disk.nvme0c0n1.out‚ 5763.79296875 disk.dm-0.in‚ 427.8046875 disk.dm-0.out‚ 5763.79296875 disk.drbd1260.out‚0.03125 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚141809 disk.nvme0n1.in‚ 427.8046875" disk.nvme0n1.out‚ 5763.79296875 disk.sda.in‚0 disk.sda.out‚7.484375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚85.12110710144043% disk./.usageGB‚1172.0634155273438! disk.nvme0c0n1.in‚ 427.8046875 disk.sda3.in‚0 disk.sda3.out‚7.484375 disk.dm-1.in‚0 disk.dm-1.out‚7.484375 disk.drbd1260.in‚0 proc.cpu.threads‚17( proc.memory.availableMB‚ 2437.7578125" proc.memory.rssMB‚ 2842.8203125* proc.memory.percent‚17.351198196411133 cpu‚22.997664091368204 ~Α€‚(:† ÛÓèÔÔ†¶­ gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.86' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽëóÇU‚(:Ê ÛÓèÔ¹¬ñ¯ disk.nvme0n1.in‚ 541.9921875 disk.dm-0.in‚ 541.9921875' proc.memory.availableMB‚ 2439.671875* proc.memory.percent‚17.351198196411133 proc.cpu.threads‚17 network.recv‚146129" disk.nvme0n1.out‚ 6244.76171875 disk.sda1.out‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2842.8203125 cpu‚23.092440831395454 disk.dm-0.out‚ 6244.76171875 disk.dm-1.out‚7.828125 disk.drbd1260.in‚0 disk.sda.out‚7.828125 disk.sda2.in‚0$ memory_percent‚85.10942459106445 network.sent‚329985! disk.nvme0c0n1.in‚ 541.9921875 disk.sda3.in‚0 disk.sda3.out‚7.828125 disk.dm-1.in‚0 disk.drbd1260.out‚0.03125 disk.sda.in‚0 disk.sda1.in‚0) disk./.usagePercent‚78.65973164102385% disk./.usageGB‚1172.0868453979492$ disk.nvme0c0n1.out‚ 6244.76171875ó—j“‚‚(:ˆ êÓèÔµ¥¢° gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.436' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z¿%Geƒ‚(:Ú êÓèÔé홳 proc.cpu.threads‚17) disk./.usagePercent‚78.65972933695605% disk.nvme0n1.out‚6629.75048828125 disk.sda.out‚ 8.3515625' disk.nvme0c0n1.out‚6629.75048828125" proc.memory.rssMB‚ 2842.8203125 network.sent‚335320 network.recv‚150200% disk./.usageGB‚1172.0868110656738 disk.dm-0.in‚ 541.9921875" disk.dm-0.out‚6629.75048828125( proc.memory.availableMB‚ 2439.6640625* proc.memory.percent‚17.351198196411133 cpu‚23.178683897784566 disk.sda3.in‚0 disk.sda3.out‚ 8.3515625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.03515625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 541.9921875 disk.nvme0n1.in‚ 541.9921875 disk.dm-1.in‚0 disk.dm-1.out‚ 8.3515625 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚85.10947227478027&ô2ð0„+ _wandbruntime‚272  _runtime‚272/]¤Æ…À ' lr-AdamW/pg1‚0.00016316648879640162 ' lr-AdamW/pg2‚0.00016316648879640162 trainer/global_step‚249 % _timestamp‚1.7884881726117144e+09 _runtime‚ 272.252156029 _step‚16c5cJ栀 ' lr-AdamW/pg1‚0.00016316648879640162 ' lr-AdamW/pg2‚0.00016316648879640162 trainer/global_step‚249 % _timestamp‚1.7884881726117144e+09 _runtime‚ 272.252156029 _step‚16‰é¹K0‡+ _wandbruntime‚272  _runtime‚272ʉk½Á'ˆ»O Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17197099308032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16786008047100 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 trainer/global_step‚249 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5459161 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17197889188864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16535151223792 _runtime‚ 272.253452302 _step‚17 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17197099308032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2418417 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5459161 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 250086182444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4227570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16946106490368 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 250856823308 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 251764695552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9665213 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 251764695552 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4227570 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19117200822272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16946106490368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16946124493312 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5459161 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 250086182444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4227570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16946106490368 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 250856823308 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 251764695552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9665213 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 251764695552 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4227570 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19117200822272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16946106490368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16946124493312 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17197099308032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16786008047100 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 trainer/global_step‚249 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5459161 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17197889188864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16535151223792 _runtime‚ 272.253452302 _step‚17 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17197099308032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2418417 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 HDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9725344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2428065 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5480898 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1325619 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1007 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9725344 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9703798 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 262648489984 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 252769454592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4244446 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5459354 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18930918754304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 _step‚18 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17265803993088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1326163 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19193591184384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227224576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 251997563392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3754228 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1036 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18930942694400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19193363959808 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17013806429696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 J>DeviceStatsMon¸&¥ÅÇitor.on_train_batch_end/inactive_split.all.freed‚3753682 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 203284480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16601200620192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16601217659552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17013806429696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4244446 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 251087322322 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17266593887232 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 252769454592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16853075622946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4244446 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 251087322322 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17266593887232 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 252769454592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16853075622946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9725344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2428065 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5480898 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1325619 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1007 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9725344 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9703798 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 262648489984 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 252769454592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4244446 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5459354 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18930918754304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 _step‚18 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17265803993088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1326163 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19193591184384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227224576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 251997563392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3754228 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1036 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18930942694400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19193363959808 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17013806429696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3753682 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 203284480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16601200620192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16601217659552'„Ä0+ _wandbruntime‚273  _runtime‚273èQaQŽË /loss/FRt1/detection_loss‚3.4143760534158596 2loss/SRqqt2/assignment_loss‚2.6004154975597675 1loss/SRqqt3/assignment_loss‚3.322355432510376 1loss/FBt1/assignment_loss‚0.27986289194775865 /loss/FBt3/detection_loss‚1.6774107966934149 epoch‚0 1loss/SRqqt4/assignment_loss‚3.259984090924263 1loss/SRqqt4/detection_loss‚1.7105346976693057 .loss/FRt2/detection_loss‚3.868808019161224 /loss/FRt3/assignment_loss‚7.341004451115926 2loss/SRqqt1/assignment_loss‚3.3687788090615904 $ _timestamp‚1.788488173514346e+09 /loss/FRt4/assignment_loss‚7.385301010281432 .loss/FRt4/detection_loss‚3.237209320944898 .loss/FBt2/detection_loss‚19.83361578732729 &loss/total_loss‚101.64456443609026 trainer/global_step‚249 1loss/SRqqt2/detection_loss‚2.5286696878763344 .loss/FBt1/detection_loss‚8.206298432852092 _runtime‚ 273.154534966 1loss/SRqqt1/detection_loss‚2.3596987113145724 .loss/FRt1/assignment_loss‚7.24121371905009 0loss/FRt2/assignment_loss‚7.2134082317352295 0loss/FBt2/assignment_loss‚0.5275932853692211 .loss/FRt3/detection_loss‚3.873841192750704 1loss/SRqqt3/detection_loss‚3.2500089029967785 .loss/FBt4/detection_loss‚4.133727305051353 _step‚19 0loss/FBt3/assignment_loss‚0.7422053890350122 0loss/FBt4/assignment_loss‚0.2682427194457786k*žIUÏ 1loss/SRqqt3/detection_loss‚3.2500089029967785 .loss/FBt4/detection_loss‚4.133727305051353 _step‚19 0loss/FBt3/assignment_loss‚0.7422053890350122 0loss/FBt4/assignment_loss‚0.2682427194457786 /loss/FRt1/detection_loss‚3.4143760534158596 2loss/SRqqt2/assignment_loss‚2.6004154975597675 1loss/SRqqt3/assignment_loss‚3.322355432510376 1loss/FBt1/assignment_loss‚0.27986289194775865 /loss/FBt3/detection_loss‚1.6774107966934149 epoch‚0 1loss/SRqqt4/assignment_loss‚3.259984090924263 1loss/SRqqt4/detection_loss‚1.7105346976693057 .loss/FRt2/detection_loss‚3.868808019161224 /loss/FRt3/assignment_loss‚7.341004451115926 2loss/SRqqt1/assignment_loss‚3.3687788090615904 $ _timestamp‚1.788488173514346e+09 /loss/FRt4/assignment_loss‚7.385301010281432 .loss/FRt4/detection_loss‚3.237209320944898 .loss/FBt2/detection_loss‚19.83361578732729 &loss/total_loss‚101.64456443609026 trainer/global_step‚249 1loss/SRqqt2/detection_loss‚2.5286696878763344 .loss/FBt1/detection_loss‚8.206298432852092 _runtime‚ 273.154534966 1loss/SRqqt1/detection_loss‚2.3596987113145724 .loss/FRt1/assignment_loss‚7.24121371905009 0loss/FRt2/assignment_loss‚7.2134082317352295 0loss/FBt2/assignment_loss‚0.5275932853692211 .loss/FRt3/detection_loss‚3.873841192750704è$¬=“‚(:ˆ ùÓèÔñ͵° gpu.0.gpu‚41.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.673' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡¨å`e‘‚(:Ú ùÓèÔ‰÷ï´ disk.dm-1.out‚ 8.9296875 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚155318% disk./.usageGB‚1172.0868110656738' disk.nvme0c0n1.out‚6917.36376953125" disk.dm-0.out‚6917.36376953125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.03515625 disk.sda.in‚0$ memory_percent‚85.11044979095459) disk./.usagePercent‚78.65972933695605 disk.nvme0n1.in‚ 541.9921875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2439.50390625* proc.memory.percent‚17.351198196411133 cpu‚23.25668181978102! disk.nvme0c0n1.in‚ 541.9921875 disk.dm-0.in‚ 541.9921875 disk.sda3.out‚ 8.9296875 disk.sda.out‚ 8.9296875 disk.sda1.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2842.8203125 network.sent‚342021% disk.nvme0n1.out‚6917.36376953125TUj7ž’‚(:“ ˆÔèÔÝ梭 gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚118.471' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.694200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?xŸe“‚(:Ú ˆÔèÔèùó¯ disk.sda1.in‚0 disk.dm-0.in‚ 541.9921875) proc.memory.availableMB‚ 2439.48046875) disk./.usagePercent‚78.65345715236717 disk.sda3.in‚0 disk.sda3.out‚ 9.4296875 disk.dm-1.out‚ 9.4296875! disk.nvme0c0n1.in‚ 541.9921875% disk.nvme0n1.out‚7111.81494140625 cpu‚23.328060077987683 proc.cpu.threads‚17 network.sent‚391788 network.recv‚161060 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚7111.81494140625$ memory_percent‚85.11059284210205# proc.memory.rssMB‚ 2842.82421875 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 541.9921875" disk.dm-0.out‚7111.81494140625* proc.memory.percent‚17.351222038269043$ disk./.usageGB‚1171.993350982666 disk.drbd1260.out‚ 0.0390625 disk.sda.out‚ 9.4296875:ª¿‘”‚(:† —ÔèÔÀÝŒ³ gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚175.33' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.066 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^5qj•‚(:ß —ÔèÔွ¸' disk.nvme0c0n1.out‚7538.80419921875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 network.sent‚399305( disk./.usagePercent‚78.6514057639953% disk./.usageGB‚1171.9627838134766 disk.dm-0.in‚ 675.49609375 disk.sda3.in‚0 disk.sda3.out‚ 9.79296875 disk.dm-1.out‚ 9.79296875 disk.sda.out‚ 9.79296875" disk.nvme0c0n1.in‚ 675.49609375 disk.nvme0n1.in‚ 675.49609375% disk.nvme0n1.out‚7538.80419921875" disk.dm-0.out‚7538.80419921875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 2439.7109375 network.recv‚164875 disk.drbd1260.out‚ 0.0390625 disk.sda1.in‚0$ memory_percent‚85.10918617248535# proc.memory.rssMB‚ 2842.82421875* proc.memory.percent‚17.351222038269043 cpu‚23.396784090950238 proc.cpu.threads‚17â˜Z0–+ _wandbruntime‚325  _runtime‚325(©ŽÆ—À ' lr-AdamW/pg1‚0.00019593084397640195 ' lr-AdamW/pg2‚0.00019593084397640195 trainer/global_step‚299 % _timestamp‚1.7884882259362676e+09 _runtime‚ 325.576860501 _step‚20jãÆÊ˜Ä ' lr-AdamW/pg1‚0.00019593084397640195 ' lr-AdamW/pg2‚0.00019593084397640195 trainer/global_step‚299 % _timestamp‚1.7884882259362676e+09 _runtime‚ 325.576860501 _step‚202ùn'0™+ _wandbruntime‚325  _runtime‚325‚Ñ,Å'š¿O IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20632347974656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2900817 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20331121459712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 300156767552 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6548063 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2900815 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22622987550720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20331121459712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20331103456768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20139400420208 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 302016395776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5071270 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19838455972432 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5071272 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚19838473011792 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20331103456768 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11597815 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20632347974656 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 301244517888 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4487572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1586755 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 313689095680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6526545 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11619335 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884882259375439e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22623011490816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 300927408416 _step‚21 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5071272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220946432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1586230 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22936676646400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20138612739984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1009 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4487045 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1038 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5071270 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 302016395776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚299 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 _runtime‚ 325.577996853 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 301244517888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6526545 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 _runtime‚ 325.577996853 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 301244517888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6526545 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20632347974656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2900817 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20331121459712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 300156767552 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6548063 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2900815 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22622987550720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20331121459712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20331103456768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20139400420208 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 302016395776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5071270 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19838455972432 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5071272 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚19838473011792 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20331103456768 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11597815 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20632347974656 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 301244517888 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4487572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1586755 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 313689095680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6526545 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11619335 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884882259375439e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22623011490816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 300927408416 _step‚21 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5071272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220946432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1586230 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22936676646400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20138612739984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648‘Ý 0œ+ _wandbruntime‚326  _runtime‚326Ç 4uÏ&ÉM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 303021154816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11636400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225127424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11636400 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19904539447552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19904522408192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20701842553856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23013067336192 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 314932630016 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884882268619058e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4502068 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20701842553856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20205680315622 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11657946 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚20398803396096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 302249263616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 314731442688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6548256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23013292463616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 _runtime‚ 326.503048242 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1009Zš7Ó&žÍM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19904539447552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19904522408192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20701842553856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23013067336192 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 314932630016 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884882268619058e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4502068 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20701842553856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20205680315622 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11657946 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚20398803396096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 302249263616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 314731442688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6548256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23013292463616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 _runtime‚ 326.503048242 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1009 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 303021154816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11636400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225127424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11636400 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BV±0Ÿ+ _wandbruntime‚326  _runtime‚326‘èQ Ë /loss/FRt3/assignment_loss‚6.689843917715138 trainer/global_step‚299 _step‚23 /loss/FRt2/assignment_loss‚6.839280913857853 0loss/FBt3/assignment_loss‚0.7425338313361862 0loss/SRqqt4/detection_loss‚1.460841244252942 /loss/FBt4/detection_loss‚3.0015858501022636 0loss/SRqqt2/detection_loss‚3.902182114869356 _runtime‚ 326.504938585 /loss/FRt4/assignment_loss‚6.697707748413086 .loss/FBt3/detection_loss‚4.745327220596135 2loss/SRqqt3/assignment_loss‚2.9059670008145844 0loss/FBt1/assignment_loss‚0.4028791309196095 1loss/FBt2/assignment_loss‚0.21654785782338876 /loss/FRt3/detection_loss‚2.5962447579307804 .loss/FRt4/detection_loss‚2.853858887878331 %loss/total_loss‚87.30578944731425 /loss/FRt1/assignment_loss‚7.164583484331767 0loss/FBt4/assignment_loss‚0.2551127087408108 /loss/FRt2/detection_loss‚3.0098111980101643 2loss/SRqqt1/assignment_loss‚3.1953356757996576 2loss/SRqqt4/assignment_loss‚2.8470824773495016 .loss/FRt1/detection_loss‚4.161396051446597 1loss/SRqqt1/detection_loss‚1.3882805106381604 0loss/SRqqt3/detection_loss‚2.088667106513794 .loss/FBt2/detection_loss‚9.797680807113647 1loss/SRqqt2/assignment_loss‚2.857113781571388 /loss/FBt1/detection_loss‚7.4859251692891124 epoch‚0 % _timestamp‚1.7884882268646898e+09bw¥"U¡Ï .loss/FBt3/detection_loss‚4.745327220596135 2loss/SRqqt3/assignment_loss‚2.9059670008145844 0loss/FBt1/assignment_loss‚0.4028791309196095 1loss/FBt2/assignment_loss‚0.21654785782338876 /loss/FRt3/detection_loss‚2.5962447579307804 .loss/FRt4/detection_loss‚2.853858887878331 %loss/total_loss‚87.30578944731425 /loss/FRt1/assignment_loss‚7.164583484331767 0loss/FBt4/assignment_loss‚0.2551127087408108 /loss/FRt2/detection_loss‚3.0098111980101643 2loss/SRqqt1/assignment_loss‚3.1953356757996576 2loss/SRqqt4/assignment_loss‚2.8470824773495016 .loss/FRt1/detection_loss‚4.161396051446597 1loss/SRqqt1/detection_loss‚1.3882805106381604 0loss/SRqqt3/detection_loss‚2.088667106513794 .loss/FBt2/detection_loss‚9.797680807113647 1loss/SRqqt2/assignment_loss‚2.857113781571388 /loss/FBt1/detection_loss‚7.4859251692891124 epoch‚0 % _timestamp‚1.7884882268646898e+09 /loss/FRt3/assignment_loss‚6.689843917715138 trainer/global_step‚299 _step‚23 /loss/FRt2/assignment_loss‚6.839280913857853 0loss/FBt3/assignment_loss‚0.7425338313361862 0loss/SRqqt4/detection_loss‚1.460841244252942 /loss/FBt4/detection_loss‚3.0015858501022636 0loss/SRqqt2/detection_loss‚3.902182114869356 _runtime‚ 326.504938585 /loss/FRt4/assignment_loss‚6.697707748413086§…Öt“¢‚(:ˆ ¦ÔèÔ³Úþ¯ gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>C‚ /£‚(:ß ¦ÔèÔ‚—˲' proc.memory.availableMB‚ 2439.421875 network.recv‚168896 disk.nvme0n1.in‚ 675.49609375$ disk.nvme0n1.out‚8008.5361328125 disk.dm-0.in‚ 675.49609375! disk.dm-0.out‚8008.5361328125 disk.sda.out‚ 10.43359375 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 675.49609375) disk./.usagePercent‚78.63480367539981 disk.dm-1.out‚ 10.43359375 disk.drbd1260.in‚0 disk.sda2.in‚0& disk.nvme0c0n1.out‚8008.5361328125# memory_percent‚85.1109504699707* proc.memory.percent‚17.351222038269043 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.04296875% disk./.usageGB‚1171.7154006958008 disk.sda3.out‚ 10.43359375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2842.82421875 cpu‚23.458541504149704 n=$ñ;etwork.sent‚402863 disk.sda3.in‚0 disk.sda.in‚0Sïsž¤‚(:“ µÔèÔŠ§‚¯ gpu.0.gpu‚22.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚132.956' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.591199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p?æšj¥‚(:ß µÔèԪ黲 disk.nvme0n1.in‚ 675.49609375! disk.dm-0.out‚8500.2431640625( proc.memory.availableMB‚ 2439.3828125 network.recv‚174964 disk.drbd1260.in‚0& disk.nvme0c0n1.out‚8500.2431640625$ disk.nvme0n1.out‚8500.2431640625* proc.memory.percent‚17.351222038269043 proc.cpu.threads‚17 disk.sda1.out‚0 network.sent‚453199% disk./.usageGB‚1170.9033699035645 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 675.49609375 disk.drbd1260.out‚ 0.04296875 disk.sda.out‚ 10.81640625 disk.sda2.in‚0# memory_percent‚85.1111888885498# proc.memory.rssMB‚ 2842.82421875 cpu‚23.51587170243722 disk.sda1.in‚0) disk./.usagePercent‚78.58030760759361 disk.sda3.out‚ 10.81640625 disk.dm-1.out‚ 10.81640625" disk.nvme0c0n1.in‚ 675.49609375D Xõ“¦‚(:ˆ ÄÔèÔ»âõ¯ gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø™X‘]§‚(:Ò ÄÔèÔæíÖ² disk.nvme0n1.in‚ 675.49609375 disk.sda3.out‚11.28125 disk.drbd1260.out‚ 0.05078125! proc.memory.rssMB‚ 2842.828125* proc.memory.percent‚17.351245880126953 proc.cpu.threads‚17 network.sent‚459582) disk./.usagePercent‚78.58030504751827% disk./.usageGB‚1170.9033317565918" disk.nvme0c0n1.in‚ 675.49609375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.out‚11.28125 disk.sda1.in‚0$ disk.nvme0n1.out‚8764.1494140625 disk.dm-1.out‚11.28125 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚85.11428833007812$ proc.memory.availableMB‚2438.875 disk.dm-0.in‚ 675.49609375! disk.dm-0.out‚8764.1494140625 disk.sda.in‚0 cpu‚23.569255940992864 network.recv‚179905& disk.nvme0c0n1.out‚8764.1494140625ƒ;0¨+ _wandbruntime‚373  _runtime‚373÷~†óÆ©À trainer/global_step‚349 % _timestamp‚1.7884882739492505e+09 _runtime‚ 373.589919191 _step‚24 ' lr-AdamW/pg1‚0.00022869519915640229 ' lr-AdamW/pg2‚0.00022869519915640229 0ü滀 ' lr-AdamW/pg2‚0.00022869519915640229 trainer/global_step‚349 % _timestamp‚1.7884882739492505e+09 _runtime‚ 373.589919191 _step‚24 ' lr-AdamW/pg1‚0.00022869519915640229»ž°´0«+ _wandbruntime‚373  _runtime‚373ì]Å'¬¿O WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23716100423168 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 351496218112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1855374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23492005113092 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 352268096000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5238036 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 366055763456 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24067596641280 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5914972 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26390404689920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 _runtime‚ 373.590981754 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26390428630016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1854821 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26756683496960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 % _timestamp‚1.7884882739504836e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 351496218112 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24067596641280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 366254866944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13530417 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223043584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24068386522112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚25 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3383215 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3383217 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5914972 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7615447 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 352268096000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13551937 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7615447 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5914970 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23141777760432 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 350227352660 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7636965 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 199103488 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5238591 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚26756460453376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24067596641280 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5914972 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26390404689920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 _runtime‚ 373.590981754 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26390428630016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1854821 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26756683496960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 % _timestamp‚1.7884882739504836e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 351496218112 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24067596641280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 366254866944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13530417 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223043584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24068386522112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚25 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3383215 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3383217 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5914972 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7615447 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 352268096000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13551937 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7615447 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5914970 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23141777760432 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 350227352660 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7636965 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 199103488ã w0®+ _wandbruntime‚374  _runtime‚374 l‹3Ä#¯ÈM MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1021 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5931844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7637158 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5931844 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13569002 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5931846 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23783818365440 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5931846 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26465776972800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26465753032704 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7658702 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 352500963840 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5253636 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 _runtime‚ 374.45140055 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13569002 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23207861235552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1050 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24137091220480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 353272855040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3392865 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26832867892736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13590548 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13590548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 367114860032 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23559860369162 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5253094 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23559072688730 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884882748104596e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 352500963840 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3392863 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190701568 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23207844196192 _step‚26 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23783818365440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7637158 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5931844 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13569002 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5931846 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23783818365440 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5931846 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26465776972800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26465753032704 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7658702 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 352500963840 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5253636 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 _runtime‚ 374.45140055 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13569002 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23207861235552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1050 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24137091220480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 353272855040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3392865 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26832867892736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13590548 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13590548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 367114860032 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23559860369162 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5253094 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23559072688730 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884882748104596e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 352500963840 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3392863 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190701568 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23207844196192 _step‚26 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23783818365440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 _step‚29 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15463019 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5990928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27101097389568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 418696565760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 400297937768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1035 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚564 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30576542334976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 trainer/global_step‚399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1064 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8725867 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27101097389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27101115392512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3865617 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚562 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6758672 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 418512142336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 401747918336 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8704349 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27503635188736 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 401068578632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27502845307904 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 402519796224ûIp% ÁÂO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 _step‚29 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15463019 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5990928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27101097389568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 418696565760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 400297937768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1035 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚564 ]JDeviceStatsMonitorfNØ£.on_train_batch_start/inactive_split_bytes.all.allocated‚30576542334976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 trainer/global_step‚399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1064 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8725867 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27101097389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27101115392512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3865617 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚562 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6758672 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 418512142336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 401747918336 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8704349 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27503635188736 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 401068578632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27502845307904 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 402519796224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6758670 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5990364 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚26845397486200 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27101115392512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30157821829120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚26846185166424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8704349 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208363520 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3865615 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 401747918336 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30576333971456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312«.Ô¨0Â+ _wandbruntime‚425  _runtime‚425öx¾kÎ&ÃÈM [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 403524555264 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 402752664064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27572339887104 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30233170171904 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26511165984192 _runtime‚ 425.85297975 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26913252742270 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2130110 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30652946443264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8747604 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3875263 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 402069718718 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6005921 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26511183023552 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 401299077646 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6775546 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2130656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218835968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15523150 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6005373 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 419557435392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27168797328896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1035 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚27571549992960 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6775546 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 trainer/global_step‚399 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6775544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15501604 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26912465061838 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15501604 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚546 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 _step‚30 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15523150 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27168815331840 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27168815331840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488PèGÖÒ&ÄÌM XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 403524555264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 % _timestamp‚1.7884883262122796e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6005373 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 419557435392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27168797328896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1035 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚27571549992960 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6775546 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 trainer/global_step‚399 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6775544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15501604 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26912465061838 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15501604 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚546 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 _step‚30 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15523150 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27168815331840 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27168815331840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 403524555264 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 402752664064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27572339887104 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30233170171904 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26511165984192 _runtime‚ 425.85297975 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26913252742270 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2130110 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30652946443264 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8747604 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3875263 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 402069718718 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6005921 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26511183023552 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 401299077646 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6775546 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2130656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218835968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15523150 ODeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30486112358912 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6742410 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33925238968320 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9814769 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30938883855360 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30938093974528 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7602370 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30486112358912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30938883855360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 450368522876 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30486094355968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4348015 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 471315623936 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9814769 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 452771496448 _runtime‚ 478.708582255 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17417141 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6741848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800CüZºÉ'ÓÃO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30938093974528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚30486094355968 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 452771496448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 451999618560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30486112358912 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6742410 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33925238968320 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9814769 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30938883855360 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30938093974528 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7602370 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30486112358912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30938883855360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 450368522876 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30486094355968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4348015 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 471315623936 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9814769 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 452771496448 _runtime‚ 478.708582255 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17417141 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6741848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 _step‚33 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9793251 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7602372 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34396372265984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376!·Âš0Ô+ _wandbruntime‚479  _runtime‚479ù~çÏ&ÕÉM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 453776255488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31006798659584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17455752 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2399256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1035 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227224576 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34000611251200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17455752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30553812298240 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6756919 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30266645115378 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9814962 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34472978751488 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 472164215808 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 451369662754 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30553812298240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30265857434946 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7619246 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29814487772192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 452140303826 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6757496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34000587311104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4357663 _step‚34 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 % _timestamp‚1.7884883800347424e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2399831 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277/9Ô·ÖÍM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29814487772192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 452140303826 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6757496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34000587311104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4357663 _step‚34 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 % _timestamp‚1.7884883800347424e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2399831 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 453776255488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31006798659584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17455752 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2399256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1035 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227224576 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34000611251200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17455752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30553812298240 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6756919 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30266645115378 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9814962 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34472978751488 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 472164215808 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 451369662754 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30553812298240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30265857434946 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7619246"ߟ!0×+ _wandbruntime‚479  _runtime‚479Kã>±OØÉ 2loss/SRqqt3/assignment_loss‚1.8688178333605445 0loss/SRqqt4/assignment_loss‚3.13472027128393 %loss/total_loss‚77.16476350216422 trainer/global_step‚449 2loss/SRqqt1/assignment_loss‚3.4173983573913573 0loss/FBt1/assignment_loss‚0.4045820374849427 .loss/FRt1/detection_loss‚3.777320738994714 0loss/SRqqt1/detection_loss‚4.154055178165436 epoch‚0 1loss/FBt3/assignment_loss‚0.35263008157151887 /loss/FRt3/detection_loss‚2.1604755181972295 /loss/FRt2/assignment_loss‚6.676331081390381 /loss/FRt4/assignment_loss‚6.701174054827009 0loss/SRqqt2/detection_loss‚2.828607221444448 /loss/FBt4/detection_loss‚1.9094726861266826 _runtime‚ 479.677315731 /loss/FRt1/assignment_loss‚7.082328839735552 /loss/FRt3/assignment_loss‚4.381806141709628 0loss/FBt4/assignment_loss‚0.4997945476264139 .loss/FBt3/detection_loss‚4.278995205055583 % _timestamp‚1.7884883800370991e+09 _step‚35 1loss/FBt2/assignment_loss‚0.34239401099226313 1loss/SRqqt3/detection_loss‚1.5053099518598512 0loss/SRqqt4/detection_loss‚3.931040240056587 /loss/FBt1/detection_loss‚2.1229028526161398 1loss/SRqqt2/assignment_loss‚2.977784643570582 .loss/FRt2/detection_loss‚3.630814740657806 .loss/FRt4/detection_loss‚4.764604314097336 .loss/FBt2/detection_loss‚4.261402953948293 Š56SÙÍ 1loss/SRqqt2/assignment_loss‚2.977784643570582 .loss/FRt2/detection_loss‚3.630814740657806 .loss/FRt4/detection_loss‚4.764604314097336 .loss/FBt2/detection_loss‚4.261402953948293 2loss/SRqqt3/assignment_loss‚1.8688178333605445 0loss/SRqqt4/assignment_loss‚3.13472027128393 %loss/total_loss‚77.16476350216422 trainer/global_step‚449 2loss/SRqqt1/assignment_loss‚3.4173983573913573 0loss/FBt1/assignment_loss‚0.4045820374849427 .loss/FRt1/detection_loss‚3.777320738994714 0loss/SRqqt1/detection_loss‚4.154055178165436 epoch‚0 1loss/FBt3/assignment_loss‚0.35263008157151887 /loss/FRt3/detection_loss‚2.1604755181972295 /loss/FRt2/assignment_loss‚6.676331081390381 /loss/FRt4/assignment_loss‚6.701174054827009 0loss/SRqqt2/detection_loss‚2.828607221444448 /loss/FBt4/detection_loss‚1.9094726861266826 _runtime‚ 479.677315731 /loss/FRt1/assignment_loss‚7.082328839735552 /loss/FRt3/assignment_loss‚4.381806141709628 0loss/FBt4/assignment_loss‚0.4997945476264139 .loss/FBt3/detection_loss‚4.278995205055583 % _timestamp‚1.7884883800370991e+09 _step‚35 1loss/FBt2/assignment_loss‚0.34239401099226313 1loss/SRqqt3/detection_loss‚1.5053099518598512 0loss/SRqqt4/detection_loss‚3.931040240056587 /loss/FBt1/detection_loss‚2.1229028526161398#VÝö”žÚ‚(:“ ¼ÕèÔ„´ç­ gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.149' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.629799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰^Û‚(:Ó ¼ÕèÔ¥ñ°° disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 969.5546875$ disk./.usageGB‚1171.239387512207 disk.nvme0n1.in‚ 969.5546875 disk.sda3.out‚ 15.30078125 disk.dm-1.out‚ 15.30078125 disk.drbd1260.out‚0.0625 disk.sda.out‚ 15.30078125$ memory_percent‚85.11490821838379 cpu‚23.899568555661187 network.recv‚216963) disk./.usagePercent‚78.60285803124717" disk.nvme0n1.out‚ 11067.4453125 disk.drbd1260.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2842.86328125* proc.memory.percent‚17.351460456848145 disk.dm-0.in‚ 969.5546875 disk.sda2.in‚0$ disk.nvme0c0n1.out‚ 11067.4453125( proc.memory.availableMB‚ 2438.7734375 proc.cpu.threads‚17 network.sent‚593713 disk.dm-0.out‚ 11067.4453125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.in‚0?ˆ0^Ü‚(:Ó ËÕèÔÈæ° disk.sda2.in‚0! disk.nvme0c0n1.in‚ 969.5546875! disk.nvme0n1.out‚ 11252.828125$ memory_percent‚85.11028289794922# proc.memory.rssMB‚ 2842.86328125 cpu‚23.930458373140905 network.sent‚642885 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.06640625 disk.sda1.in‚0# disk.nvme0c0n1.out‚ 11252.828125& proc.memory.availableMB‚ 2439.53125* proc.memory.percent‚17.351460456848145% disk./.usageGB‚1171.7394180297852 disk.dm-0.out‚ 11252.828125 disk.sda3.out‚ 15.70703125 disk.sda.in‚0 disk.sda.out‚ 15.70703125 proc.cpu.threads‚17 network.recv‚221935) disk./.usagePercent‚78.63641549883539 disk.dm-0.in‚ 969.5546875 disk.sda3.in‚0 disk.dm-1.out‚ 15.70703125 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 969.5546875 disk.dm-1.in‚0Rs¤“Ý‚(:ˆ ËÕèÔ¤ìç± gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.433' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ù›“Þ‚(:ˆ ÚÕèÔµÿê¬ gpu.0.gpu‚41.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.127' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆöºkß‚(:à ÚÕèÔüº¬¯ disk.sda.in‚0 disk.sda.out‚ 16.16796875( disk.nvme0c0n1.out‚11594.94580078125 disk.dm-0.in‚ 1074.15625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 1074.15625 disk.nvme0n1.in‚ 1074.15625# disk.dm-0.out‚11594.94580078125 disk.sda3.out‚ 16.16796875$ memory_percent‚85.11509895324707 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 16.16796875( proc.memory.availableMB‚ 2438.7421875) disk./.usagePercent‚78.63638989808197 disk.drbd1260.out‚ 0.06640625& disk.nvme0n1.out‚11594.94580078125# proc.memory.rssMB‚ 2842.86328125* proc.memory.percent‚17.351460456848145 cpu‚23.95982324724291 proc.cpu.threads‚17 network.sent‚648397 network.recv‚225674% disk./.usageGB‚1171.7390365600586ʾ̒à‚(:‡ éÕèÔËò¹± gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.779' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.9558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓÎ"bá‚(:× éÕèÔ¹–úµ# disk.dm-0.out‚12063.39111328125$ memory_percent‚85.11223793029785 disk.sda1.in‚0) disk./.usagePercent‚78.64469478249272 disk.sda3.out‚16.53125 disk.dm-1.in‚0 disk.dm-0.in‚ 1074.15625" proc.memory.rssMB‚ 2842.8671875 cpu‚23.988363129140232 proc.cpu.threads‚17 network.recv‚231003& disk.nvme0n1.out‚12063.39111328125( proc.memory.availableMB‚ 2439.2109375* proc.memory.percent‚17.351484298706055 disk.drbd1260.out‚ 0.07421875 disk.sda.out‚16.53125 disk.sda1.out‚0 network.sent‚655057% disk./.usageGB‚1171.8627853393555 disk.sda3.in‚0 disk.dm-1.out‚16.53125 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 1074.15625( disk.nvme0c0n1.out‚12063.39111328125 disk.nvme0n1.in‚ 1074.15625õò=î0â+ _wandbruntime‚527  _runtime‚527IŒáCÆãÀ ' lr-AdamW/pg1‚0.00032698826469640326 ' lr-AdamW/pg2‚0.00032698826469640326 trainer/global_step‚499 % _timestamp‚1.7884884282546597e+09 _runtime‚ 527.895170396 _step‚36]pÍ)ÊäÄ _step‚36 ' lr-AdamW/pg1‚0.00032698826469640326 ' lr-AdamW/pg2‚0.00032698826469640326 trainer/global_step‚499 % _timestamp‚1.7884884282546597e+09 _runtime‚ 527.895170396$Ä#+0å+ _wandbruntime‚527  _runtime‚527ìN¬È'æÂO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10903671 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33051743124432 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10882153 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34374132521984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚37 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33871091322368 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚559 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4830415 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37692680047616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 % _timestamp‚1.7884884282560985e+09 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 503023196672 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 502251318784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 199103488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚33051760163792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19328223 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚34374132521984 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34373342641152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33871109325312 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7494632 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34373342641152 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 503023196672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2664215 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8446070 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7494073 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2663658 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚557 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223043584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 _runtime‚ 527.89672712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19349743Œ¿ýçÆO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33871109325312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33871091322368 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 502251318784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10882153 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4830417 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37692656107520 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8446072 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38216173029888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34373342641152 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 503023196672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2664215 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8446070 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7494073 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2663658 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚557 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223043584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 _runtime‚ 527.89672712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19349743 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10903671 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33051743124432 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10882153 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34374132521984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚37 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33871091322368 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚559 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4830415 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37692680047616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 % _timestamp‚1.7884884282560985e+09 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 503023196672 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 502251318784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 199103488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚33051760163792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19328223 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚34374132521984 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34373342641152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33871109325312 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7494632 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7509138 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34442837220352 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34442047326208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19388354 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8462944 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884884292854018e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37768004450304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38292796245504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37768028390400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10925408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 524572959232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33938809264640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2669075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218835968 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 501440247862 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19388354 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10903864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33938791261696 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 504027955712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¡WìÚÖ&êÐM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19388354 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10903864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33938791261696 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 504027955712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7509138 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34442837220352 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34442047326208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19388354 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8462944 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884884292854018e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37768004450304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38292796245504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37768028390400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10925408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 524572959232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33938809264640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2669075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218835968 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 501440247862&e%N©0ë+ _wandbruntime‚528  _runtime‚528¨^¯GTìÎ 1loss/SRqqt3/detection_loss‚1.6579538215714764 _step‚39 trainer/global_step‚499 % _timestamp‚1.7884884292884996e+09 .loss/FBt1/detection_loss‚9.862539432265542 /loss/FBt2/detection_loss‚3.9486972424719067 epoch‚0 _runtime‚ 528.928744383 /loss/FRt1/assignment_loss‚6.862438716386494 2loss/SRqqt3/assignment_loss‚1.7372468813232718 0loss/FBt1/assignment_loss‚0.7851471735802161 0loss/FBt4/assignment_loss‚0.2849655511018066 /loss/FRt2/detection_loss‚3.5005982173116585 0loss/SRqqt4/detection_loss‚2.852297399725233 /loss/FBt4/detection_loss‚1.6950582446222524 0loss/SRqqt2/detection_loss‚3.614890922199596 2loss/SRqqt2/assignment_loss‚2.4542953660993865 /loss/FRt4/detection_loss‚2.8852041906780665 .loss/FRt2/assignment_loss‚6.21150353080348 2loss/SRqqt1/assignment_loss‚3.1274842635179176 2loss/SRqqt4/assignment_loss‚3.1300401119958785 /loss/FRt3/detection_loss‚1.7228779726943304 %loss/total_loss‚87.96948102268667 /loss/FBt3/assignment_loss‚0.959887683391571 1loss/SRqqt1/detection_loss‚2.9773930448752184 /loss/FBt3/detection_loss‚14.383603811264038 /loss/FRt3/assignment_loss‚4.890790839527929 0loss/FRt4/assignment_loss‚5.0251461293962265 1loss/FBt2/assignment_loss‚0.34637034998309596 .loss/FRt1/detection_loss‚3.053050125900068-š.?XíÒ 2loss/SRqqt2/assignment_loss‚2.4542953660993865 /loss/FRt4/detection_loss‚2.8852041906780665 .loss/FRt2/assignment_loss‚6.21150353080348 2loss/SRqqt1/assignment_loss‚3.1274842635179176 2loss/SRqqt4/assignment_loss‚3.1300401119958785 /loss/FRt3/detection_loss‚1.7228779726943304 %loss/total_loss‚87.96948102268667 /loss/FBt3/assignment_loss‚0.959887683391571 1loss/SRqqt1/detection_loss‚2.9773930448752184 /loss/FBt3/detection_loss‚14.383603811264038 /loss/FRt3/assignment_loss‚4.890790839527929 0loss/FRt4/assignment_loss‚5.0251461293962265 1loss/FBt2/assignment_loss‚0.34637034998309596 .loss/FRt1/detection_loss‚3.053050125900068 1loss/SRqqt3/detection_loss‚1.6579538215714764 _step‚39 trainer/global_step‚499 % _timestamp‚1.7884884292884996e+09 .loss/FBt1/detection_loss‚9.862539432265542 /loss/FBt2/detection_loss‚3.9486972424719067 epoch‚0 _runtime‚ 528.928744383 /loss/FRt1/assignment_loss‚6.862438716386494 2loss/SRqqt3/assignment_loss‚1.7372468813232718 0loss/FBt1/assignment_loss‚0.7851471735802161 0loss/FBt4/assignment_loss‚0.2849655511018066 /loss/FRt2/detection_loss‚3.5005982173116585 0loss/SRqqt4/detection_loss‚2.852297399725233 /loss/FBt4/detection_loss‚1.6950582446222524 0loss/SRqqt2/detection_loss‚3.614890922199596'9áWÿžî‚(:“ øÕèÔù‰ã¯ gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚128.127' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.625400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›‚_ï‚(:Ô øÕèÔ럲$ disk./.usageGB‚1171.987075805664 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 1074.15625& disk.nvme0n1.out‚12393.67236328125# disk.dm-0.out‚12393.67236328125 disk.drbd1260.in‚0 disk.sda.out‚16.9375) proc.memory.availableMB‚ 2438.44140625 network.sent‚661737 network.recv‚236639 disk.sda1.out‚0( disk.nvme0c0n1.out‚12393.67236328125 disk.nvme0n1.in‚ 1074.15625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.07421875 proc.cpu.threads‚17) disk./.usagePercent‚78.65303601997334 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚16.9375 disk.sda.in‚0$ memory_percent‚85.11693477630615* proc.memory.percent‚17.351484298706055 disk.dm-0.in‚ 1074.15625 disk.dm-1.out‚16.9375" proc.memory.rssMB‚ 2842.8671875 cpu‚24.015143655583877 :4“ð‚(:ˆ ‡ÖèÔ’ð® gpu.0.gpu‚41.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œmËfñ‚(:Û ‡ÖèԵܘ² disk.drbd1260.out‚0.078125 disk.sda.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚12615.58544921875# memory_percent‚85.1139783859253) proc.memory.availableMB‚ 2438.92578125 cpu‚24.042341120140748 disk.sda3.in‚0 disk.dm-1.out‚ 17.3828125 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 1074.15625 proc.cpu.threads‚17 network.recv‚241624) disk./.usagePercent‚78.65303473993566 disk.sda3.out‚ 17.3828125 disk.drbd1260.in‚0 disk.sda.out‚ 17.3828125 disk.sda2.out‚0( disk.nvme0c0n1.out‚12615.58544921875" proc.memory.rssMB‚ 2842.8671875 network.sent‚710799% disk./.usageGB‚1171.9870567321777 disk.dm-1.in‚0 disk.nvme0n1.in‚ 1074.15625 disk.dm-0.in‚ 1074.15625# disk.dm-0.out‚12615.58544921875* proc.memory.percent‚17.351484298706055jÊÀú‘ò‚(:† –ÖèÔ¶ªÙ® gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚161.41' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.282 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jì /nó‚(:ã –ÖèÔŸŽó± disk.sda3.out‚ 17.80859375 network.sent‚715946 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚13030.52294921875& disk.nvme0n1.out‚13030.52294921875 disk.dm-1.out‚ 17.80859375' proc.memory.availableMB‚ 2438.921875" proc.memory.rssMB‚ 2842.8671875* proc.memory.percent‚17.351484298706055( disk./.usagePercent‚78.6514610616227$ disk./.usageGB‚1171.963607788086 disk.sda.out‚ 17.80859375 disk.dm-1.in‚0 disk.drbd1260.out‚0.078125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0# memory_percent‚85.1140022277832 cpu‚24.069454269152335 proc.cpu.threads‚17 network.recv‚245118 disk.sda.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1222.43359375! disk.nvme0n1.in‚ 1222.43359375 disk.dm-0.in‚ 1222.43359375# disk.dm-0.out‚13030.52294921875Ƈøžô‚(:“ ¥ÖèÔëá­ gpu.0.gpu‚19.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.149' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.629799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´EVtõ‚(:é ¥ÖèÔ ˆÖ°" proc.memory.rssMB‚ 2842.8671875* proc.memory.percent‚17.351484298706055 network.recv‚250350( disk./.usagePercent‚78.6535710757199 disk.sda3.out‚ 18.22265625 disk.sda1.in‚0 network.sent‚722271( disk.nvme0c0n1.out‚13180.10693359375& disk.nvme0n1.out‚13180.10693359375 disk.dm-0.in‚ 1222.43359375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 18.22265625$ memory_percent‚85.11407375335693% disk./.usageGB‚1171.9950485229492 disk.sda2.out‚0! disk.nvme0n1.in‚ 1222.43359375# disk.dm-0.out‚13180.10693359375 disk.sda1.out‚0) proc.memory.availableMB‚ 2438.91015625 cpu‚24.092405325528755 proc.cpu.threads‚17 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1222.43359375 disk.sda3.in‚0 disk.dm-1.out‚ 18.22265625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.08203125øtu´ö+ NÜëq*_wandbruntime‚589  _runtime‚589LdJºÂ÷¼ & lr-AdamW/pg2‚0.0003597526198764036 trainer/global_step‚549 $ _timestamp‚1.788488489366459e+09 _runtime‚ 589.00726401 _step‚40 & lr-AdamW/pg1‚0.0003597526198764036Š9bÉÆøÀ _runtime‚ 589.00726401 _step‚40 & lr-AdamW/pg1‚0.0003597526198764036 & lr-AdamW/pg2‚0.0003597526198764036 trainer/global_step‚549 $ _timestamp‚1.788488489366459e+09(Àb†W0ù+ _wandbruntime‚589  _runtime‚589þ<ø-É'úÃO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21260825 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37808591307776 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37809381188608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37256088288768 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 550509693092 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37808591307776 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 575992253440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚549 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36355081951792 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36355064912432 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8245961 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11992573 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9289772 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9289770 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8246522 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37256106291712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42036286446592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9289770 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11992573 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41460073246720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21282345 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 _step‚41 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197006336 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 551280333956 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 % _timestamp‚1.7884884893681304e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2933705 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 _runtime‚ 589.008963302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37256106291712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5312815 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41460097186816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36905574605524 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 552503019008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9289772 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1041 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 576189259776 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 553274896896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11971055 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 553274896896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36906362285748 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840–oÿÍ'ûÇO ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 553274896896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36906362285748 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21260825 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37808591307776 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37809381188608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37256088288768 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 550509693092 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37808591307776 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 575992253440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚549 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36355081951792 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36355064912432 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8245961 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11992573 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9289772 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9289770 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8246522 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37256106291712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42036286446592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9289770 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11992573 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41460073246720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21282345 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 _step‚41 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197006336 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 551280333956 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 % _timestamp‚1.7884884893681304e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2933705 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 _runtime‚ 589.008963302 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37256106291712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5312815 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41460097186816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36905574605524 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 552503019008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9289772 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1041 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 576189259776 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 553274896896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11971055 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617)pBj@0ü+ _wandbruntime‚589  _runtime‚589—$2Ò&ýÌM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21320956 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37878085886976 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8260907 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2938444 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37877295992832 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21320956 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5322465 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 577038441472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9306644 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9306646 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36972642181162 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 553507764736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194895872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 _runtime‚ 589.970111422 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 551510832970 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 554279655936 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 553507764736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36421131348192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 $ _timestamp‚1.788488490328732e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1070 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12014310 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9306646 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944–ñ¦GþÐM _runtime‚ 589.970111422 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 551510832970 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 554279655936 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 553507764736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36421131348192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 $ _timestamp‚1.788488490328732e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1070 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12014310 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9306646 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21320956 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37878085886976 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8260907 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2938444 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37877295992832 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21320956 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5322465 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 577038441472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9306644 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9306646 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36972642181162 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 553507764736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194895872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800*ß_áû0ÿ+ _wandbruntime‚589  _runtime‚589² …OT€Î .loss/FBt2/detection_loss‚4.371953862054007 % _timestamp‚1.7884884903313715e+09 /loss/FRt1/assignment_loss‚6.201058604500511 /loss/FBt1/detection_loss‚3.6145715713500977 /loss/FBt3/detection_loss‚30.624584555625916 /loss/FRt3/assignment_loss‚7.154307978493827 /loss/FRt4/detection_loss‚1.9261801545049104 /loss/FBt4/detection_loss‚0.7648435233379758 0loss/FBt1/assignment_loss‚0.4548591123566439 0loss/SRqqt1/detection_loss‚3.022435784339905 0loss/SRqqt3/detection_loss‚2.101040524741014 1loss/SRqqt4/detection_loss‚1.7602284274274302 %loss/total_loss‚96.50962576391443 trainer/global_step‚549 _step‚43 /loss/FRt2/assignment_loss‚5.622469139099121 1loss/SRqqt3/assignment_loss‚2.624218120091203 0loss/FBt3/assignment_loss‚0.5284723043441772 /loss/FRt1/detection_loss‚3.1804098252094155 _runtime‚ 589.972051708 0loss/FRt4/assignment_loss‚3.5903018494726906 1loss/SRqqt2/assignment_loss‚2.535164002917315 1loss/FBt4/assignment_loss‚0.34441924906462795 .loss/FRt3/detection_loss‚3.408819847873279 epoch‚0 2loss/SRqqt4/assignment_loss‚2.9820533260222404 1loss/FBt2/assignment_loss‚0.15722320397336428 .loss/FRt2/detection_loss‚3.248735484480858 2loss/SRqqt1/assignment_loss‚3.2648820355534554 1loss/SRqqt2/detection_loss‚3.0263932770804356Œ¬yÊXÒ /loss/FRt2/assignment_loss‚5.622469139099121 1loss/SRqqt3/assignment_loss‚2.624218120091203 0loss/FBt3/assignment_loss‚0.5284723043441772 /loss/FRt1/detection_loss‚3.1804098252094155 _runtime‚ 589.972051708 0loss/FRt4/assignment_loss‚3.5903018494726906 1loss/SRqqt2/assignment_loss‚2.535164002917315 1loss/FBt4/assignment_loss‚0.34441924906462795 .loss/FRt3/detection_loss‚3.408819847873279 epoch‚0 2loss/SRqqt4/assignment_loss‚2.9820533260222404 1loss/FBt2/assignment_loss‚0.15722320397336428 .loss/FRt2/detection_loss‚3.248735484480858 2loss/SRqqt1/assignment_loss‚3.2648820355534554 1loss/SRqqt2/detection_loss‚3.0263932770804356 .loss/FBt2/detection_loss‚4.371953862054007 % _timestamp‚1.7884884903313715e+09 /loss/FRt1/assignment_loss‚6.201058604500511 /loss/FBt1/detection_loss‚3.6145715713500977 /loss/FBt3/detection_loss‚30.624584555625916 /loss/FRt3/assignment_loss‚7.154307978493827 /loss/FRt4/detection_loss‚1.9261801545049104 /loss/FBt4/detection_loss‚0.7648435233379758 0loss/FBt1/assignment_loss‚0.4548591123566439 0loss/SRqqt1/detection_loss‚3.022435784339905 0loss/SRqqt3/detection_loss‚2.101040524741014 1loss/SRqqt4/detection_loss‚1.7602284274274302 %loss/total_loss‚96.50962576391443 trainer/global_step‚549 _step‚43+Ùg½ž‚‚(:“ ´ÖèÔ‘Û‹® gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.243' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.048599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠÍÈkƒ‚(:à ´ÖèÔšï°$ disk./.usageGB‚1171.995018005371 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚13518.201171875 proc.cpu.threads‚17) disk./.usagePercent‚78.65356902765963 disk.dm-1.out‚ 18.6171875# disk.nvme0c0n1.in‚ 1222.43359375& disk.nvme0c0n1.out‚13518.201171875 cpu‚24.115399163113494 network.sent‚728447 disk.sda3.out‚ 18.6171875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 1222.43359375! disk.dm-0.out‚13518.201171875 network.recv‚254939 disk.drbd1260.out‚ 0.08203125 disk.sda.out‚ 18.6171875! disk.nvme0n1.in‚ 1222.43359375$ memory_percent‚85.11602878570557) proc.memory.availableMB‚ 2438.58984375" proc.memory.rssMB‚ 2842.8671875* proc.memory.percent‚17.351484298706055ÛÌÓž„‚(:“ ÃÖèÔÓ«±­ gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.645' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.729000000000006 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùz¶o…‚(:ä ÃÖèÔñ¢›°# disk.nvme0c0n1.in‚ 1222.43359375( disk.nvme0c0n1.out‚13824.64208984375 disk.sda3.out‚ 19.1484375 disk.dm-1.out‚ 19.1484375 disk.drbd1260.in‚0 disk.sda.in‚0) disk./.usagePercent‚78.65356646758428% disk./.usageGB‚1171.9949798583984! disk.nvme0n1.in‚ 1222.43359375& disk.nvme0n1.out‚13824.64208984375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2842.8671875 disk.dm-0.in‚ 1222.43359375 disk.dm-1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚13824.64208984375 disk.sda3.in‚0 disk.drbd1260.out‚ 0.0859375 disk.sda.out‚ 19.1484375$ memory_percent‚85.11176109313965( proc.memory.availableMB‚ 2439.2890625* proc.memory.percent‚17.351484298706055 cpu‚24.13699633293879 network.sent‚777805 network.recv‚260434Œˆ÷’†‚(:‡ ÒÖèÔЛ£° gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚34.0 gpu.0.powerWatts‚131.649' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ugó p‡‚(:å ÒÖèÔ€›Ý² disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚85.11183261871338) proc.memory.availableMB‚ 2439.27734375* proc.memory.percent‚17.351484298706055 disk.sda1.in‚0 disk.sda3.out‚ 19.45703125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0859375 disk.sda.out‚ 19.45703125 disk.dm-0.in‚ 1328.11328125" proc.memory.rssMB‚ 2842.8671875 proc.cpu.threads‚17 disk.sda2.in‚0' disk.nvme0c0n1.out‚13998.4287109375" disk.dm-0.out‚13998.4287109375 cpu‚24.15751095583951 disk.sda1.out‚0 network.sent‚783009) disk./.usagePercent‚78.66198732340806% disk./.usageGB‚1172.1204566955566 disk.dm-1.out‚ 19.45703125# disk.nvme0c0n1.in‚ 1328.11328125! disk.nvme0n1.in‚ 1328.11328125% disk.nvme0n1.out‚13998.4287109375 network.recv‚263990 disk.sda3.in‚0 disk.dm-1.in‚0œ)«|0ˆ+ _wandbruntime‚641  _runtime‚641…|hJĉ¾ & lr-AdamW/pg1‚0.0003925169750564039 & lr-AdamW/pg2‚0.0003925169750564039 trainer/global_step‚599 % _timestamp‚1.7884885417008767e+09 _runtime‚ 641.341255838 _step‚44Y½rRÈŠÂ trainer/global_step‚599 % _timestamp‚1.7884885417008767e+09 _runtime‚ 641.341255838 _step‚44 & lr-AdamW/pg1‚0.0003925169750564039 & lr-AdamW/pg2‚0.0003925169750564039,34 Ç0‹+ _wandbruntime‚641  _runtime‚641Îö4Ì'ŒÆO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13081475 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45227514326016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10133472 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41243839974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218849280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5795217 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39658403739792 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45227490385920 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 628633637376 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41244629855232 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40641103258112 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8998172 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41244629855232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10133470 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 603526597120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3203519 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3202957 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 602754719232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194909184 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40259754658856 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 600580278200 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41243839974400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23193427 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1041 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 _runtime‚ 641.342442292 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23214947 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 601350919064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10133472 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41243839974400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218849280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5795217 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39658403739792 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45227490385920 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 628633637376 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41244629855232 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40641103258112 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8998172 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41244629855232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10133470 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 603526597120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3203519 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3202957 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 602754719232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194909184 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40259754658856 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 600580278200 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41243839974400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1070 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23193427 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39-Óê0Ž+ _wandbruntime‚642  _runtime‚642|Ö¤×&ÑM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196993024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 601581418078 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45932519264768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45302862668800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40708785194496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1041 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10150344 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 604531356160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884885425829089e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10150346 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚599 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23253558 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41313334553600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39724453136192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23232012 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41312544659456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10150344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 603759464960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 _step‚46 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41313334553600 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23253558 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23232012 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13103212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚558 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40326822234702 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10150346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220933120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9013272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39724470175552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1070 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40708803197440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3208409 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 603759464960 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45932740197888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45302838728704 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 629877529088¬BkÛ&ÕM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10150346 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚599 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23253558 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41313334553600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39724453136192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23232012 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41312544659456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10150344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 603759464960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 _step‚46 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41313334553600 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23253558 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23232012 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13103212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚558 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40326822234702 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10150346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220933120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9013272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39724470175552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1070 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40708803197440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3208409 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 603759464960 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45932740197888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45302838728704 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 629877529088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196993024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 601581418078 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45932519264768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45302862668800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40708785194496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1041 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10150344 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 604531356160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884885425829089e+09.U ‚ó0‘+ _wandbruntime‚642  _runtime‚642RSQ U’Ï 2loss/SRqqt1/assignment_loss‚3.1908157893589566 /loss/FRt2/detection_loss‚2.6683960490756564 /loss/FBt4/detection_loss‚1.0479984602361219 % _timestamp‚1.7884885425855105e+09 1loss/SRqqt2/assignment_loss‚2.469979584837953 0loss/FBt4/assignment_loss‚0.2654652627568375 /loss/FRt1/detection_loss‚2.6664491565454576 /loss/FRt4/detection_loss‚1.9558299406078354 trainer/global_step‚599 _runtime‚ 642.225749612 /loss/FRt1/assignment_loss‚6.272692351114182 2loss/SRqqt4/assignment_loss‚2.1595320328102483 0loss/FBt1/assignment_loss‚0.6370515205365207 1loss/FBt2/assignment_loss‚0.33392425403484693 /loss/FRt3/detection_loss‚2.6856251173424273 .loss/FBt1/detection_loss‚3.443297702641714 .loss/FBt3/detection_loss‚5.383446753025055 0loss/FRt2/assignment_loss‚5.9563290807935925 1loss/SRqqt3/assignment_loss‚2.582283747692903 %loss/total_loss‚66.27814170724167 epoch‚0 _step‚47 0loss/FRt4/assignment_loss‚2.3334851871276725 1loss/SRqqt2/detection_loss‚3.0733573416868847 1loss/SRqqt4/detection_loss‚1.0966544286771254 /loss/FRt3/assignment_loss‚5.193901900975209 0loss/FBt3/assignment_loss‚0.6451382264494896 1loss/SRqqt1/detection_loss‚3.2638206680615744 0loss/SRqqt3/detection_loss‚1.659262020347847 .loss/FBt2/detection_loss‚5.293405130505562Á͘éY“Ó /loss/FRt2/detection_loss‚2.6683960490756564 /loss/FBt4/detection_loss‚1.0479984602361219 % _timestamp‚1.7884885425855105e+09 1loss/SRqqt2/assignment_loss‚2.469979584837953 0loss/FBt4/assignment_loss‚0.2654652627568375 /loss/FRt1/detection_loss‚2.6664491565454576 /loss/FRt4/detection_loss‚1.9558299406078354 trainer/global_step‚599 _runtime‚ 642.225749612 /loss/FRt1/assignment_loss‚6.272692351114182 2loss/SRqqt4/assignment_loss‚2.1595320328102483 0loss/FBt1/assignment_loss‚0.6370515205365207 1loss/FBt2/assignment_loss‚0.33392425403484693 /loss/FRt3/detection_loss‚2.6856251173424273 .loss/FBt1/detection_loss‚3.443297702641714 .loss/FBt3/detection_loss‚5.383446753025055 0loss/FRt2/assignment_loss‚5.9563290807935925 1loss/SRqqt3/assignment_loss‚2.582283747692903 %loss/total_loss‚66.27814170724167 epoch‚0 _step‚47 0loss/FRt4/assignment_loss‚2.3334851871276725 1loss/SRqqt2/detection_loss‚3.0733573416868847 1loss/SRqqt4/detection_loss‚1.0966544286771254 /loss/FRt3/assignment_loss‚5.193901900975209 0loss/FBt3/assignment_loss‚0.6451382264494896 1loss/SRqqt1/detection_loss‚3.2638206680615744 0loss/SRqqt3/detection_loss‚1.659262020347847 .loss/FBt2/detection_loss‚5.293405130505562 2loss/SRqqt1/assignment_loss‚3.1908157893589566/¨V°Ûž”‚(:“ áÖèÔÔÅÍ® gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.918' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.183600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ÉŸ l•‚(:á áÖèÔ·òž±% disk.nvme0c0n1.out‚14558.12890625 disk.sda3.in‚0 disk.dm-1.out‚ 19.88671875 disk.drbd1260.out‚ 0.08984375) proc.memory.availableMB‚ 2439.23828125 cpu‚24.177105323792137 network.recv‚268701 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1328.11328125# disk.nvme0n1.out‚14558.12890625 disk.dm-0.in‚ 1328.11328125 disk.dm-0.out‚14558.12890625" proc.memory.rssMB‚ 2842.8671875 proc.cpu.threads‚17 network.sent‚802240 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1328.11328125 disk.sda3.out‚ 19.88671875 disk.dm-1.in‚0 disk.drbd1260.in‚0) disk./.usagePercent‚78.66195148235326% disk./.usageGB‚1172.1199226379395 disk.sda.in‚0 disk.sda.out‚ 19.88671875$ memory_percent‚85.11207103729248* proc.memory.percent‚17.351484298706055ñÁ'Bž–‚(:“ ðÖèÔåºÞ® gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.689' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.537799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d¤Òãe—‚(:Ú ðÖèԚγ) proc.memory.availableMB‚ 2439.23046875" proc.memory.rssMB‚ 2842.8671875 cpu‚24.196355776290705) disk./.usagePercent‚78.66719170057162 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 1328.11328125 disk.dm-0.out‚14804.66015625 network.sent‚838919% disk./.usageGB‚1172.1980056762695 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.out‚0# disk.nvme0n1.out‚14804.66015625 disk.dm-0.in‚ 1328.11328125# memory_percent‚85.1121187210083 disk.drbd1260.out‚ 0.08984375 disk.sda.out‚ 20.359375% disk.nvme0c0n1.out‚14804.66015625* proc.memory.percent‚17.351484298706055 proc.cpu.threads‚17 network.recv‚274267 disk.sda3.out‚ 20.359375 disk.dm-1.out‚ 20.359375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 1328.11328125‘¡Ð¦“˜‚(:ˆ ÿÖèÔ‘Ô­ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚125.199' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0398 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äãi™‚(:Þ ÿÖèÔêîÚ¯ disk.dm-1.out‚ 20.859375 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 1328.11328125 disk.dm-0.in‚ 1328.11328125 disk.sda3.in‚0$ memory_percent‚85.11366844177246* proc.memory.percent‚17.351484298706055 cpu‚24.21452966031453 proc.cpu.threads‚17) disk./.usagePercent‚78.67656413640016 disk.sda.out‚ 20.859375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0.09375( proc.memory.availableMB‚ 2438.9765625" proc.memory.rssMB‚ 2842.8671875 network.sent‚843374 network.recv‚277522 disk.sda1.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1328.11328125& disk.nvme0n1.out‚15392.99560546875 disk.sda3.out‚ 20.859375$ disk./.usageGB‚1172.337661743164 disk.sda.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚15392.99560546875# disk.dm-0.out‚15392.99560546875ƒoÌ‘š‚(:† Ž×èÔæËÞ¬ gpu.0.gpu‚4.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.945 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(VŽ@a›‚(:Ö Ž×èÔضÀ¯ disk.sda3.out‚21.28125 disk.drbd1260.in‚0 disk.drbd1260.out‚0.09375 disk.sda1.in‚0( proc.memory.availableMB‚ 2438.9765625 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚21.28125! disk.nvme0c0n1.in‚ 1367.546875( disk.nvme0c0n1.out‚15700.57373046875" proc.memory.rssMB‚ 2842.8671875* proc.memory.percent‚17.351484298706055 disk.dm-1.in‚0 disk.dm-1.out‚21.28125 disk.nvme0n1.in‚ 1367.546875& disk.nvme0n1.out‚15700.57373046875 cpu‚24.23146296501742 proc.cpu.threads‚17% disk./.usageGB‚1171.4255905151367 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1367.546875# disk.dm-0.out‚15700.57373046875$ memory_percent‚85.11366844177246 network.sent‚848437 network.recv‚280939) disk./.usagePercent‚78.61535427100851‰ªn0œ+ _wandbruntime‚694  _runtime‚694Loé¼ trainer/global_step‚649 $ _timestamp‚1.788488594412776e+09 _runtime‚ 694.05359661 _step‚48 & lr-AdamW/pg1‚0.0004252813302364042 & lr-AdamW/pg2‚0.0004252813302364042DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14170377 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 681423515136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44679878521856 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3473118 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 % _timestamp‚1.7884885944135957e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42961708488432 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚558 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25147549 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 _runtime‚ 694.054393632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1070 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44679088641024 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9750735 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205394944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44026082221568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229335040 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49676125645312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44679088641024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44026082221568 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44679088641024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44026082221568 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14170377 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 681423515136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44679878521856 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3473118 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 % _timestamp‚1.7884885944135957e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42961708488432 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚558 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25147549 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 _runtime‚ 694.054393632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1070 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44679088641024 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9750735 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205394944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚440260822215681;'\0¢+ _wandbruntime‚694  _runtime‚694˜kDlÕ&£ÏM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10994046 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14192114 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1041 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10994044 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9765213 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43027791963552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43027774924192 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10994046 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43679426927378 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44747793326080 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 _step‚50 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44093782160896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14170570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25186160 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44748583220224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14192114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 204155904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 654011165184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚558 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9765773 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25164614 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49752547459584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 654783056384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1070 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10994044 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚649 _runtime‚ 694.98007705 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9765213 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43027791963552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43027774924192 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10994046 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43679426927378 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44747793326080 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 _step‚50 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44093782160896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14170570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25186160 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44748583220224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14192114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 204155904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 654011165184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚558 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9765773 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25164614 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49752547459584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 654783056384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1070 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10994044 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚649 _runtime‚ 694.98007705 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10994046 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14192114 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1041 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10994044 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 % _timestamp‚1.7884886486616302e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 700721448416 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 703258119680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 703258119680 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 734064635904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48114337307648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11820872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11820870 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 % _timestamp‚1.7884886486616302e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 700721448416 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 703258119680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 703258119680 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 734064635904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48114337307648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11820872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11820870 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6769663 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15281016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1074 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53572823879680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 201187328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47478779127296 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 705034756608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 705034756608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15281016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225127424 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46331096712192 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 704262865408 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15259472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15259472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10517441 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11837744 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47478779127296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27118762 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47032819300486 _step‚54 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27118762 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27097216 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 701722588294 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53572598752256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52837696947200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47033606980918 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46331113751552 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10517441 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11837744 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47478779127296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27118762 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47032819300486 _step‚54 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27118762 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27097216 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 701722588294 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53572598752256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52837696947200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47033606980918 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46331113751552 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6769663 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15281016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1074 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53572823879680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 201187328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47478779127296 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 705034756608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 705034756608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15281016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225127424 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46331096712192 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 704262865408 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15259472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚152594726mÏ“¾0·+ _wandbruntime‚749  _runtime‚749·é‘S¸Í 2loss/SRqqt3/assignment_loss‚2.4784217918360674 /loss/FBt1/detection_loss‚2.6138549271751854 .loss/FBt3/detection_loss‚6.733431458473206 /loss/FRt2/assignment_loss‚6.293229639530182 0loss/FBt1/assignment_loss‚0.6790650185002425 1loss/SRqqt1/detection_loss‚3.0021090068315206 /loss/FBt2/detection_loss‚3.9414691507816313 trainer/global_step‚699 _step‚55 0loss/FBt4/assignment_loss‚0.2915866306200704 /loss/FRt2/detection_loss‚3.4068706209460893 /loss/FRt1/detection_loss‚3.8694538010491266 .loss/FRt3/assignment_loss‚4.83829966097167 2loss/SRqqt2/assignment_loss‚1.9064264122177572 .loss/FRt4/detection_loss‚2.018388863585212 .loss/FRt1/assignment_loss‚6.04580905702379 2loss/FBt3/assignment_loss‚0.010515992529690266 .loss/FRt3/detection_loss‚2.049460168589245 .loss/FBt4/detection_loss‚0.885580814124406 epoch‚0 1loss/FBt2/assignment_loss‚0.26939254949490227 /loss/FRt4/assignment_loss‚2.337987633184953 1loss/SRqqt1/assignment_loss‚3.056355037187275 1loss/SRqqt4/assignment_loss‚1.826512277217173 1loss/SRqqt4/detection_loss‚1.0617292073623437 &loss/total_loss‚63.892530592759556 1loss/SRqqt2/detection_loss‚2.7966986061895596 1loss/SRqqt3/detection_loss‚1.4798822673382583 $ _timestamp‚1.788488649627345e+09 _runtime‚ 749.268597458mþUëW¹Ñ 2loss/SRqqt3/assignment_loss‚2.4784217918360674 /loss/FBt1/detection_loss‚2.6138549271751854 .loss/FBt3/detection_loss‚6.733431458473206 /loss/FRt2/assignment_loss‚6.293229639530182 0loss/FBt1/assignment_loss‚0.6790650185002425 1loss/SRqqt1/detection_loss‚3.0021090068315206 /loss/FBt2/detection_loss‚3.9414691507816313 trainer/global_step‚699 _step‚55 0loss/FBt4/assignment_loss‚0.2915866306200704 /loss/FRt2/detection_loss‚3.4068706209460893 /loss/FRt1/detection_loss‚3.8694538010491266 .loss/FRt3/assignment_loss‚4.83829966097167 2loss/SRqqt2/assignment_loss‚1.9064264122177572 .loss/FRt4/detection_loss‚2.018388863585212 .loss/FRt1/assignment_loss‚6.04580905702379 2loss/FBt3/assignment_loss‚0.010515992529690266 .loss/FRt3/detection_loss‚2.049460168589245 .loss/FBt4/detection_loss‚0.885580814124406 epoch‚0 1loss/FBt2/assignment_loss‚0.26939254949490227 /loss/FRt4/assignment_loss‚2.337987633184953 1loss/SRqqt1/assignment_loss‚3.056355037187275 1loss/SRqqt4/assignment_loss‚1.826512277217173 1loss/SRqqt4/detection_loss‚1.0617292073623437 &loss/total_loss‚63.892530592759556 1loss/SRqqt2/detection_loss‚2.7966986061895596 1loss/SRqqt3/detection_loss‚1.4798822673382583 $ _timestamp‚1.788488649627345e+09 _runtime‚ 749.2685974587öüꉒº‚(:‡ Ê×èÔý²Þ­ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚177.925' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~¾½o»‚(:ä Ê×èÔ¯©±$ memory_percent‚85.11431217193604) disk./.usagePercent‚78.62220170452684 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 1439.1171875# proc.memory.rssMB‚ 2842.87109375 network.sent‚914364 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0( disk.nvme0c0n1.out‚17193.28857421875 disk.nvme0n1.in‚ 1439.1171875) proc.memory.availableMB‚ 2438.87109375* proc.memory.percent‚17.351508140563965 network.recv‚298037% disk./.usageGB‚1171.5276222229004 disk.dm-0.in‚ 1439.1171875 disk.sda.out‚ 23.1640625 cpu‚24.296231284692045 proc.cpu.threads‚17# disk.dm-0.out‚17193.28857421875 disk.sda3.out‚ 23.1640625 disk.dm-1.out‚ 23.1640625 disk.drbd1260.out‚ 0.1015625 disk.sda1.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚17193.28857421875@ü?“¼‚(:ˆ Ù×èÔ¹“ ­ gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.909' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.5818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ²†øu½‚(:ê Ù×èÔ¤¬Â¯# disk.dm-0.out‚17317.18701171875 disk.drbd1260.in‚0 disk.sda.in‚0 network.sent‚964251 disk.sda2.out‚0 disk.sda3.out‚ 23.76171875 disk.dm-1.out‚ 23.76171875 disk.drbd1260.out‚ 0.10546875 disk.sda.out‚ 23.76171875 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1600.37109375 disk.sda3.in‚0( proc.memory.availableMB‚ 2438.8515625* proc.memory.percent‚17.351508140563965% disk./.usageGB‚1171.8825988769531! disk.nvme0n1.in‚ 1600.37109375 disk.dm-0.in‚ 1600.37109375 disk.dm-1.in‚0$ memory_percent‚85.11443138122559# proc.memory.rssMB‚ 2842.87109375 cpu‚24.310601054588595 proc.cpu.threads‚17 network.recv‚303785) disk./.usagePercent‚78.64602448562556 disk.sda2.in‚0( disk.nvme0c0n1.out‚17317.18701171875& disk.nvme0n1.out‚17317.18701171875ø·ª“¾‚(:ˆ è×èÔ£šÎ­ gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.286' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù£u¿‚(:ê è×èÔƒÔ•± network.sent‚971997% disk./.usageGB‚1171.9378776550293 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 1600.37109375# disk.dm-0.out‚17672.97607421875 proc.cpu.threads‚17 network.recv‚308154( disk.nvme0c0n1.out‚17672.97607421875! disk.nvme0n1.in‚ 1600.37109375 disk.sda3.in‚0 disk.sda3.out‚ 24.18359375 disk.dm-1.in‚0 disk.dm-1.out‚ 24.18359375 disk.drbd1260.out‚ 0.10546875) disk./.usagePercent‚78.64973429080425 disk.sda1.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1600.37109375& disk.nvme0n1.out‚17672.97607421875 disk.sda.out‚ 24.18359375) proc.memory.availableMB‚ 2438.85546875 cpu‚24.32455938412897 disk.drbd1260.in‚0 disk.sda.in‚0$ memory_percent‚85.11440753936768# proc.memory.rssMB‚ 2842.87109375* proc.memory.percent‚17.351508140563965R`€©“À‚(:ˆ ÷×èÔª²¹® gpu.0.gpu‚20.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.856' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.1712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j)¤tÁ‚(:é ÷×èÔ”­†± disk.sda.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1600.37109375! disk.nvme0n1.in‚ 1600.37109375 cpu‚24.338142834467522 proc.cpu.threads‚17 disk.drbd1260.out‚0.109375 disk.sda.out‚ 24.58984375% disk./.usageGB‚1171.9143905639648# disk.dm-0.out‚18033.97412109375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚18033.97412109375 disk.sda3.in‚0 disk.sda3.out‚ 24.58984375 disk.dm-1.out‚ 24.58984375 disk.sda1.out‚0( disk.nvme0c0n1.out‚18033.97412109375) proc.memory.availableMB‚ 2439.09765625* proc.memory.percent‚17.351508140563965 network.sent‚975225 network.recv‚311829) disk./.usagePercent‚78.64815805241592 disk.dm-0.in‚ 1600.37109375 disk.sda2.out‚0$ memory_percent‚85.11292934417725# proc.memory.rssMB‚ 2842.87109375VÈxM0Â+ _wandbruntime‚799  _runtime‚799Ž¹‰YÃý $ _timestamp‚1.788488699468963e+09 _runtime‚ 799.109402717 _step‚56 & lr-AdamW/pg1‚0.0004908100405964049 & lr-AdamW/pg2‚0.0004908100405964049 trainer/global_step‚749>Wœ ÇÄÁ trainer/global_step‚749 $ _timestamp‚1.788488699468963e+09 _runtime‚ 799.109402717 _step‚56 & lr-AdamW/pg1‚0.0004908100405964049 & lr-AdamW/pg2‚0.00049081004059640498,þfˆ0Å+ _wandbruntime‚799  _runtime‚799”-” ÆÆO UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51550375855104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16326663 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28991233 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51549585974272 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51550375855104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 754281697792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197006336 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚50796094157312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 _runtime‚ 799.111576473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚50796076154368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDevi¤‚9â8ceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 754281697792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1074 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 786743267840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56529765743616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29012753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 " _timestamp‚1.7884886994707e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 753509819904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12664570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12664572 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56529741803520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚749 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 750792033524 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51549585974272 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 753509819904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚50796076154368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16326663 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49568352064432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 _step‚57 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 786546261504 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚50319144097956 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50319931778180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÐY·ÍÐ'ÇÊO XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50319931778180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51550375855104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16326663 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28991233 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51549585974272 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51550375855104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 754281697792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197006336 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚50796094157312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 _runtime‚ 799.111576473 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚50796076154368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 754281697792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1074 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 786743267840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56529765743616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29012753 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 " _timestamp‚1.7884886994707e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 753509819904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12664570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12664572 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56529741803520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚749 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 750792033524 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51549585974272 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 753509819904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚50796076154368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16326663 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49568352064432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 _step‚57 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 786546261504 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚503191440979569á2«0È+ _wandbruntime‚800  _runtime‚800ƒ#šÙ&ÉÓM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50863776093696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56605114086400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51619080553472 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49634435539552 _runtime‚ 800.073206763 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 754514565632 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 754514565632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 752563814474 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 787583569408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50863794096640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29029818 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 _step‚58 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57392673715712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29051364 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4017291 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50863776093696 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4017866 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16369918 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7252063 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚575 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16348374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11269354 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1045 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51618290659328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12681446 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12681444 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29051364 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚50386211673594 % _timestamp‚1.7884887004318066e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚50863794096640 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 755286456832 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 787784756736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 201187328 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49634418500192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16369918 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16348374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1074 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57392898843136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225127424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12681444 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16369918 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16348374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1074 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57392898843136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225127424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12681444 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 754514565632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 752563814474 L>DeviceStatsMonitor.on_train_batch_end/activIj0¦e_bytes.all.current‚ 789894144 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 787583569408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50863794096640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29029818 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 _step‚58 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57392673715712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29051364 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4017291 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50863776093696 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4017866 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16369918 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7252063 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚575 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16348374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11269354:N/ƒ0Ë+ _wandbruntime‚800  _runtime‚800XÌÒ /loss/FBt4/detection_loss‚0.8585171232109666 0loss/FRt4/assignment_loss‚1.8323527292201394 1loss/SRqqt3/detection_loss‚1.4160153207390807 .loss/FBt2/detection_loss‚4.221212515234948 _runtime‚ 800.075606565 .loss/FRt2/detection_loss‚2.779472702741623 epoch‚0 2loss/SRqqt4/assignment_loss‚1.5537915625742504 .loss/FBt3/detection_loss‚5.022019584973653 2loss/SRqqt1/assignment_loss‚3.0147513055801394 2loss/SRqqt3/assignment_loss‚2.0175755620002747 0loss/FBt4/assignment_loss‚0.3755082604816932 /loss/FRt4/detection_loss‚1.8194282227440883 1loss/SRqqt4/detection_loss‚1.2279094442725182 trainer/global_step‚749 0loss/FRt2/assignment_loss‚6.2499559879302975 1loss/SRqqt2/detection_loss‚2.7794120788574217 .loss/FBt1/detection_loss‚2.606133222579956 /loss/FRt1/detection_loss‚3.0168816645940146 /loss/FRt3/detection_loss‚1.8772267248067591 1loss/SRqqt1/detection_loss‚2.8244076347351075 0loss/FBt1/assignment_loss‚0.5117495095466438 1loss/FBt2/assignment_loss‚0.45281887948513033 _step‚59 &loss/total_loss‚60.246332697399836 % _timestamp‚1.7884887004353843e+09 /loss/FRt1/assignment_loss‚5.893006501374422 /loss/FRt3/assignment_loss‚3.990355980065134 2loss/SRqqt2/assignment_loss‚2.6509263018767038 0loss/FBt3/assignment_loss‚1.2549038777748744ÜL‰\ÍÖ 0loss/FRt2/assignment_loss‚6.2499559879302975 1loss/SRqqt2/detection_loss‚2.7794120788574217 .loss/FBt1/detection_loss‚2.606133222579956 /loss/FRt1/detection_loss‚3.0168816645940146 /loss/FRt3/detection_loss‚1.8772267248067591 1loss/SRqqt1/detection_loss‚2.8244076347351075 0loss/FBt1/assignment_loss‚0.5117495095466438 1loss/FBt2/assignment_loss‚0.45281887948513033 _step‚59 &loss/total_loss‚60.246332697399836 % _timestamp‚1.7884887004353843e+09 /loss/FRt1/assignment_loss‚5.893006501374422 /loss/FRt3/assignment_loss‚3.990355980065134 2loss/SRqqt2/assignment_loss‚2.6509263018767038 0loss/FBt3/assignment_loss‚1.2549038777748744 /loss/FBt4/detection_loss‚0.8585171232109666 0loss/FRt4/assignment_loss‚1.8323527292201394 1loss/SRqqt3/detection_loss‚1.4160153207390807 .loss/FBt2/detection_loss‚4.221212515234948 _runtime‚ 800.075606565 .loss/FRt2/detection_loss‚2.779472702741623 epoch‚0 2loss/SRqqt4/assignment_loss‚1.5537915625742504 .loss/FBt3/detection_loss‚5.022019584973653 2loss/SRqqt1/assignment_loss‚3.0147513055801394 2loss/SRqqt3/assignment_loss‚2.0175755620002747 0loss/FBt4/assignment_loss‚0.3755082604816932 /loss/FRt4/detection_loss‚1.8194282227440883 1loss/SRqqt4/detection_loss‚1.2279094442725182 trainer/global_step‚749;¹íç“΂(:ˆ †ØèÔŽ§õ­ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.043' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6086 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îüj'pÏ‚(:å †ØèÔš‡÷¯ disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2842.87109375* proc.memory.percent‚17.351508140563965) disk./.usagePercent‚78.68364914491026 disk.sda.out‚ 25.42578125% proc.memory.availableMB‚ 2438.5625 cpu‚24.351324332092968 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 1600.37109375( disk.nvme0c0n1.out‚18447.39990234375! disk.nvme0n1.in‚ 1600.37109375# disk.dm-0.out‚18447.39990234375 disk.dm-1.out‚ 25.42578125 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 network.sent‚980350 network.recv‚315765% disk./.usageGB‚1172.4432334899902 disk.dm-0.in‚ 1600.37109375 disk.dm-1.in‚0 disk.drbd1260.out‚0.109375$ memory_percent‚85.11619567871094& disk.nvme0n1.out‚18447.39990234375 disk.sda3.in‚0 disk.sda3.out‚ 25.42578125Õ¥Ú¾“Ђ(:ˆ •ØèÔùÚ¬± gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.779' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MŽ…¶sÑ‚(:è •ØèÔ€ôœµ disk.dm-0.in‚ 1744.5546875$ memory_percent‚85.11471748352051( proc.memory.availableMB‚ 2438.8046875# proc.memory.rssMB‚ 2842.87109375* proc.memory.percent‚17.351508140563965 network.recv‚321852) disk./.usagePercent‚78.68364786487258 disk.sda3.in‚0 disk.sda3.out‚ 25.8203125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 1744.5546875 network.sent‚ 1.03065e+06 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.11328125 disk.sda.out‚ 25.8203125 disk.sda1.out‚0( disk.nvme0c0n1.out‚18735.24658203125& disk.nvme0n1.out‚18735.24658203125# disk.dm-0.out‚18735.24658203125 disk.dm-1.out‚ 25.8203125 disk.sda1.in‚0 disk.nvme0n1.in‚ 1744.5546875 cpu‚24.363989823424443 proc.cpu.threads‚17$ disk./.usageGB‚1172.443214416504 disk.drbd1260.in‚0è«ÍžÒ‚(:“ ¤ØèÔ¦¿À® gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.181' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.236200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ´'…dÓ‚(:Ù ¤ØèÔž†§´& disk.nvme0n1.out‚19077.95361328125# disk.dm-0.out‚19077.95361328125 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.11328125 disk.sda.out‚26.25 disk.sda2.out‚0# proc.memory.rssMB‚ 2842.87109375( disk./.usagePercent‚78.6878394762307" disk.nvme0c0n1.in‚ 1744.5546875 disk.nvme0n1.in‚ 1744.5546875 disk.dm-0.in‚ 1744.5546875) proc.memory.availableMB‚ 2438.55078125* proc.memory.percent‚17.351508140563965 proc.cpu.threads‚17 network.recv‚326139( disk.nvme0c0n1.out‚19077.95361328125 disk.sda3.out‚26.25 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚85.11626720428467 network.sent‚ 1.036427e+06$ disk./.usageGB‚1172.505672454834 disk.dm-1.out‚26.25 disk.sda.in‚0 disk.sda2.in‚0 cpu‚24.37603750110668sŠÿ0Ô+ _wandbruntime‚851  _runtime‚851ó@­ÄÕ¾ & lr-AdamW/pg1‚0.0005235743957764052 & lr-AdamW/pg2‚0.0005235743957764052 trainer/global_step‚799 % _timestamp‚1.7884887522217717e+09 _runtime‚ 851.862286024 _step‚60ä„~&ÈÖ trainer/global_step‚799 % _timestamp‚1.7884887522217717e+09 _runtime‚ 851.862286024 _step‚60 & lr-AdamW/pg1‚0.0005235743957764052 & lr-AdamW/pg2‚0.0005235743957764052<°nD0×+ _wandbruntime‚851  _runtime‚851 -eÏ'ØÉO % _timestamp‚1.7884887522230716e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13508272 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53672536471064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202072064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54984834640896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17415565 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61136732697600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53673324151288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚54181091123712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17437083 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60297158942720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1074 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7724817 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61136530625536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 839549814784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 800862618632 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 trainer/global_step‚799 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54984834640896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54181091123712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17437083 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7724815 _step‚61 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 801633259496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 804533398016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12007273 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1045 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 803761520128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54181073120768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17415565 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 803761520128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4282456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54985624521728 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30945355 _runtime‚ 851.863644343 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52871673852432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0.°Ê Ó'ÙÍO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53672536471064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202072064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54984834640896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17415565 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61136732697600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53673324151288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚54181091123712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17437083 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60297158942720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1074 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7724817 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61136530625536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 839549814784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 800862618632 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 trainer/global_step‚799 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54984834640896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54181091123712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17437083 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7724815 _step‚61 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 801633259496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 804533398016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12007273 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1045 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 803761520128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54181073120768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17415565 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 803761520128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4282456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54985624521728 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30945355 _runtime‚ 851.863644343 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52871673852432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884887522230716e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13508272 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30983966 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13525144 YFDeviceStat™žU$sMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54248773060096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4287797 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 trainer/global_step‚799 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17437276 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52937757327552 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17437276 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1074 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4287258 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61212951033856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17458820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54248791063040 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 840443748352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54248773060096 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 805538157056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206253056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61213157286912 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 804766265856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 _step‚62 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7734465 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 840626061312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17458820 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30983966 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55054329220096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13525144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12021721 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 % _timestamp‚1.7884887531001074e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12022262 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 805538157056 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30962420 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 852.741167465 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664ÇMnÝ&Ü×M TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61213157286912 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 804766265856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 _step‚62 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7734465 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 840626061312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17458820 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30983966 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55054329220096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13525144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12021721 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 % _timestamp‚1.7884887531001074e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12022262 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 805538157056 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30962420 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 852.741167465 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30962420 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182312960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30983966 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13525144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54248773060096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4287797 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 trainer/global_step‚799 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17437276 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52937757327552 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17437276 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1074 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4287258 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61212951033856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17458820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54248791063040 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 840443748352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54248773060096 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 805538157056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206253056>Ãø†Q0Ý+ _wandbruntime‚852  _runtime‚852ëŽ dQÞË 2loss/SRqqt3/assignment_loss‚2.1202895073095958 0loss/FBt3/assignment_loss‚0.8035991191864014 .loss/FRt3/detection_loss‚2.005343627297517 0loss/FRt4/assignment_loss‚1.9287045732621224 .loss/FRt2/detection_loss‚2.833079380648477 /loss/FRt4/detection_loss‚1.7474641487483056 1loss/SRqqt3/detection_loss‚1.5196313162644703 1loss/SRqqt2/assignment_loss‚2.651382016284125 /loss/FRt1/detection_loss‚2.9563960870107016 1loss/SRqqt2/detection_loss‚2.5974341673510417 % _timestamp‚1.7884887531029422e+09 1loss/SRqqt1/assignment_loss‚2.472051697196784 2loss/SRqqt4/assignment_loss‚1.4551613333706672 -loss/FBt3/detection_loss‚8.13271427154541 1loss/FBt2/assignment_loss‚0.49015675616266724 0loss/SRqqt4/detection_loss‚1.125047280238225 .loss/FBt1/detection_loss‚3.647543428465724 /loss/FBt4/detection_loss‚0.9025314152751939 %loss/total_loss‚61.80242355282172 0loss/FBt4/assignment_loss‚0.1924705956043275 1loss/SRqqt1/detection_loss‚2.9496270329863936 _runtime‚ 852.743274306 /loss/FRt2/assignment_loss‚4.993474457945142 /loss/FBt1/assignment_loss‚0.787510742280574 /loss/FBt2/detection_loss‚3.3229894692247566 epoch‚0 trainer/global_step‚799 _step‚63 /loss/FRt1/assignment_loss‚5.562834739685059 /loss/FRt3/assignment_loss‚4.604986389478047åø¼UßÏ 1loss/SRqqt1/assignment_loss‚2.472051697196784 2loss/SRqqt4/assignment_loss‚1.4551613333706672 -loss/FBt3/detection_loss‚8.13271427154541 1loss/FBt2/assignment_loss‚0.49015675616266724 0loss/SRqqt4/detection_loss‚1.125047280238225 .loss/FBt1/detection_loss‚3.647543428465724 /loss/FBt4/detection_loss‚0.9025314152751939 %loss/total_loss‚61.80242355282172 0loss/FBt4/assignment_loss‚0.1924705956043275 1loss/SRqqt1/detection_loss‚2.9496270329863936 _runtime‚ 852.743274306 /loss/FRt2/assignment_loss‚4.993474457945142 /loss/FBt1/assignment_loss‚0.787510742280574 /loss/FBt2/detection_loss‚3.3229894692247566 epoch‚0 trainer/global_step‚799 _step‚63 /loss/FRt1/assignment_loss‚5.562834739685059 /loss/FRt3/assignment_loss‚4.604986389478047 2loss/SRqqt3/assignment_loss‚2.1202895073095958 0loss/FBt3/assignment_loss‚0.8035991191864014 .loss/FRt3/detection_loss‚2.005343627297517 0loss/FRt4/assignment_loss‚1.9287045732621224 .loss/FRt2/detection_loss‚2.833079380648477 /loss/FRt4/detection_loss‚1.7474641487483056 1loss/SRqqt3/detection_loss‚1.5196313162644703 1loss/SRqqt2/assignment_loss‚2.651382016284125 /loss/FRt1/detection_loss‚2.9563960870107016 1loss/SRqqt2/detection_loss‚2.5974341673510417 % _timestamp‚1.7884887531029422e+09?Ø—bŸà‚(:’ ³ØèÔÑô„® gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.098' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚33.81960000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?wá‚(:ì ³ØèÔò§ì¯% disk./.usageGB‚1172.5681533813477 disk.nvme0n1.in‚ 1744.5546875 disk.sda3.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 2439.0078125 network.sent‚ 1.041514e+06( disk.nvme0c0n1.out‚19324.87451171875 disk.sda3.out‚ 26.69921875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.351508140563965 cpu‚24.388153495690343 network.recv‚329685 disk.dm-0.in‚ 1744.5546875# disk.dm-0.out‚19324.87451171875 disk.drbd1260.in‚0 disk.sda.out‚ 26.69921875$ memory_percent‚85.11347770690918 proc.cpu.threads‚17) disk./.usagePercent‚78.69203262363402" disk.nvme0c0n1.in‚ 1744.5546875& disk.nvme0n1.out‚19324.87451171875 disk.dm-1.out‚ 26.69921875 disk.drbd1260.out‚ 0.1171875 disk.sda.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2842.87109375ôO?È“â‚(:ˆ ÂØèÔµ±é­ gpu.0.gpu‚32.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.377' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ºhã‚(:Ý ÂØèÔ¦ºû²" disk.nvme0c0n1.in‚ 1744.5546875( disk.nvme0c0n1.out‚19739.82373046875& disk.nvme0n1.out‚19739.82373046875" memory_percent‚85.113525390625 cpu‚24.39922671685486 disk.sda.in‚0 disk.sda1.out‚0 network.sent‚ 1.09219e+06 network.recv‚334027) disk./.usagePercent‚78.70042327056875% disk./.usageGB‚1172.6931800842285 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.1171875 disk.nvme0n1.in‚ 1744.5546875# disk.dm-0.out‚19739.82373046875* proc.memory.percent‚17.351508140563965 proc.cpu.threads‚17 disk.sda.out‚ 27.0546875 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 27.0546875 disk.dm-1.out‚ 27.0546875 disk.dm-0.in‚ 1744.5546875 proc.memory.availableMB‚2439# proc.memory.rssMB‚ 2842.87109375 disk.dm-1.in‚0 disk.sda2.in‚0º2½’ä‚(:‡ ÑØèÔ•¿›­ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚174.509' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ïèÜiå‚(:Þ ÑØèÔ•“Ò¯$ disk./.usageGB‚1172.944179534912' disk.nvme0c0n1.out‚20178.8662109375 disk.nvme0n1.in‚ 1906.3671875 disk.dm-1.out‚27.40625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.12109375 disk.sda.in‚0 disk.dm-0.in‚ 1906.3671875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚85.11507511138916) proc.memory.availableMB‚ 2438.74609375# proc.memory.rssMB‚ 2842.87109375 cpu‚24.41054720314415 network.recv‚338529( disk./.usagePercent‚78.7172680543067% disk.nvme0n1.out‚20178.8662109375" disk.dm-0.out‚20178.8662109375 disk.sda3.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 1906.3671875 disk.sda3.out‚27.40625 disk.dm-1.in‚0 disk.sda.out‚27.40625* proc.memory.percent‚17.351508140563965 network.sent‚ 1.09647e+06GY“æ‚(:ˆ àØèÔìÙŒ­ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.076' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.2152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jaqrç‚(:ç àØèÔú”°² disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 1906.3671875 proc.cpu.threads‚17) disk./.usagePercent‚78.71719995630258 disk.sda3.in‚0 disk.sda3.out‚ 27.9921875% disk.nvme0n1.out‚20520.7822265625* proc.memory.percent‚17.351508140563965 network.sent‚ 1.102403e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 27.9921875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.12109375 disk.sda2.in‚0( proc.memory.availableMB‚ 2438.7421875 cpu‚24.421469349489534 network.recv‚342719 disk.sda.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 1906.3671875' disk.nvme0c0n1.out‚20520.7822265625$ memory_percent‚85.11509895324707# proc.memory.rssMB‚ 2842.87109375 disk.sda.out‚ 27.9921875% disk./.usageGB‚1172.9431648254395 disk.dm-0.in‚ 1906.3671875" disk.dm-0.out‚20520.7822265625q^M0è+ _wandbruntime‚903  _runtime‚903¼äÄé¾ % _timestamp‚1.7884888043318317e+09 _runtime‚ 903.972276615 _step‚64 & lr-AdamW/pg1‚0.0005563387509564055 & lr-AdamW/pg2‚0.0005563387509564055 trainer/global_step‚849ÒyWÈê & lr-AdamW/pg1‚0.0005563387509564055 & lr-AdamW/pg2‚0.0005563387509564055 trainer/global_step‚849 % _timestamp‚1.7884888043318317e+09 _runtime‚ 903.972276615 _step‚64@ÞCö0ë+ _wandbruntime‚903  _runtime‚903yô¿ìÉO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64064600022016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 _runtime‚ 903.975089676 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32877957 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18504467 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57566088090112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56175012679792 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14351970 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58420083307520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 892041574400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4551700 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1045 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64956617656320 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57025928844172 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚564 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64956838602752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64064576081920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14351972 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 854013220352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14351970 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197006336 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18525985 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚58420873188352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_staîšort/segment.large_pool.allocated‚516 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 854785098240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 _step‚65 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58420083307520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 851703844604 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14351972 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8207217 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 892238580736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12759481 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 % _timestamp‚1.7884888043331573e+09 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 850933203740 trainer/global_step‚849 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 854785098240 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58420873188352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18504467 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57566070087168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57566070087168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57026716524396 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840\–ZÓ'íÍO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 854785098240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 _step‚65 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58420083307520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 851703844604 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14351972 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8207217 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 892238580736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12759481 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 % _timestamp‚1.7884888043331573e+09 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 850933203740 trainer/global_step‚849 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 854785098240 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58420873188352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18504467 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57566070087168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57566070087168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57026716524396 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64064600022016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 _runtime‚ 903.975089676 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32877957 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18504467 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57566088090112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56175012679792 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14351970 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58420083307520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 892041574400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4551700 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1045 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64956617656320 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57025928844172 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚564 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64956838602752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64064576081920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14351972 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 854013220352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14351970 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 197006336 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18525985 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚58420873188352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 855789857280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32916568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56241062076192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18547722 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 855789857280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18547722 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64139948364800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180215808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14368846 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64139924424704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57633770026496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 904.879193453 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14368844 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4557609 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 851934343618 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12773923 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 852704984690 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12774474 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65033249965568 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 893121384960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58489577886720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32895022 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4557060 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 855017966080 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚551 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1045 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57092996419810 _step‚661j€¬òðÖM R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57092996419810 _step‚66 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18526178 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 855789857280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32916568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56241062076192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18547722 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 855789857280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18547722 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64139948364800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180215808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14368846 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64139924424704 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57633770026496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 904.879193453 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14368844 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4557609 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 851934343618 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonit¿ûyêê or.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12773923 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 852704984690 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12774474 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65033249965568 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 893121384960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58489577886720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32895022 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4557060 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 855017966080 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚551 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1045 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0BCµ(,0ñ+ _wandbruntime‚904  _runtime‚904bg&RòÌ 1loss/SRqqt4/detection_loss‚1.5370956680991432 -loss/FBt1/detection_loss‚3.15409231884405 /loss/FBt2/detection_loss‚3.2819600462913514 _step‚67 2loss/SRqqt2/assignment_loss‚2.5596596177886513 0loss/FBt3/assignment_loss‚0.5738678475220998 .loss/FBt3/detection_loss‚3.980679988861084 /loss/FRt2/assignment_loss‚5.595469176769257 /loss/FRt3/assignment_loss‚4.659808043638865 1loss/SRqqt1/assignment_loss‚2.593698730071386 1loss/FBt4/assignment_loss‚0.26432706586915755 1loss/SRqqt3/detection_loss‚1.4566934308627757 1loss/SRqqt4/assignment_loss‚1.560068613697182 /loss/FRt1/detection_loss‚2.7422827511298946 /loss/FBt4/detection_loss‚0.9387705606989889 0loss/FRt4/assignment_loss‚1.7261060320195698 /loss/FRt1/assignment_loss‚5.666796178352542 2loss/SRqqt3/assignment_loss‚2.1704093250559597 0loss/FBt1/assignment_loss‚0.7727691680192947 trainer/global_step‚849 % _timestamp‚1.7884888052408557e+09 /loss/FRt3/detection_loss‚1.8647011466821035 _runtime‚ 904.881241733 1loss/SRqqt1/detection_loss‚2.3973207626587305 %loss/total_loss‚57.91013021876385 epoch‚0 /loss/FBt2/assignment_loss‚0.537310642004013 /loss/FRt2/detection_loss‚2.6024755090475082 /loss/FRt4/detection_loss‚2.6717918259756908 /loss/SRqqt2/detection_loss‚2.60197576880455K\BVóÐ .loss/FBt3/detection_loss‚3.980679988861084 /loss/FRt2/assignment_loss‚5.595469176769257 /loss/FRt3/assignment_loss‚4.659808043638865 1loss/SRqqt1/assignment_loss‚2.593698730071386 1loss/FBt4/assignment_loss‚0.26432706586915755 1loss/SRqqt3/detection_loss‚1.4566934308627757 1loss/SRqqt4/assignment_loss‚1.560068613697182 /loss/FRt1/detection_loss‚2.7422827511298946 /loss/FBt4/detection_loss‚0.9387705606989889 0loss/FRt4/assignment_loss‚1.7261060320195698 /loss/FRt1/assignment_loss‚5.666796178352542 2loss/SRqqt3/assignment_loss‚2.1704093250559597 0loss/FBt1/assignment_loss‚0.7727691680192947 trainer/global_step‚849 % _timestamp‚1.7884888052408557e+09 /loss/FRt3/detection_loss‚1.8647011466821035 _runtime‚ 904.881241733 1loss/SRqqt1/detection_loss‚2.3973207626587305 %loss/total_loss‚57.91013021876385 epoch‚0 /loss/FBt2/assignment_loss‚0.537310642004013 /loss/FRt2/detection_loss‚2.6024755090475082 /loss/FRt4/detection_loss‚2.6717918259756908 /loss/SRqqt2/detection_loss‚2.60197576880455 1loss/SRqqt4/detection_loss‚1.5370956680991432 -loss/FBt1/detection_loss‚3.15409231884405 /loss/FBt2/detection_loss‚3.2819600462913514 _step‚67 2loss/SRqqt2/assignment_loss‚2.5596596177886513 0loss/FBt3/assignment_loss‚0.5738678475220998CN°“æ“ô‚(:ˆ ïØèÔ¸´¦° gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚37.0 gpu.0.powerWatts‚165.356' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.0712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°B€kõ‚(:à ïØèÔ€ùв disk.sda1.out‚0% disk.nvme0c0n1.out‚21014.44140625) proc.memory.availableMB‚ 2438.66015625* proc.memory.percent‚17.351508140563965 network.recv‚346281) disk./.usagePercent‚78.71719995630258 disk.dm-1.out‚ 28.38671875 disk.sda1.in‚0 disk.nvme0n1.in‚ 1906.3671875 disk.dm-0.out‚21014.44140625 cpu‚24.431431260804448 disk.sda3.in‚0 disk.sda3.out‚ 28.38671875 disk.drbd1260.out‚0.125 disk.sda.in‚0 disk.sda.out‚ 28.38671875 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0n1.out‚21014.44140625% disk./.usageGB‚1172.9431648254395 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 1906.3671875 disk.dm-0.in‚ 1906.3671875$ memory_percent‚85.11559963226318# proc.memory.rssMB‚ 2842.87109375 proc.cpu.threads‚17 network.sent‚ 1.107522e+06 disk.drbd1260.in‚0'0•Ï“ö‚(:ˆ þØèÔ‘œï­ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.916' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9832 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°='d÷‚(:Ù þØèÔŠ†¥±# disk.nvme0n1.out‚21261.05859375 disk.sda.out‚ 28.81640625" memory_percent‚85.115647315979) proc.memory.availableMB‚ 2438.65234375 proc.cpu.threads‚17% disk./.usageGB‚1172.9431838989258" disk.nvme0c0n1.in‚ 1906.3671875 disk.nvme0n1.in‚ 1906.3671875 disk.dm-0.in‚ 1906.3671875 disk.dm-0.out‚21261.05859375 disk.sda3.in‚0 disk.sda3.out‚ 28.81640625 disk.sda.in‚0% disk.nvme0c0n1.out‚21261.05859375 disk.dm-1.out‚ 28.81640625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 proc.memory.rssMB‚2842.875 network.recv‚350679 disk.dm-1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.351531982421875 cpu‚24.441671482622276 network.sent‚ 1.156247e+06) disk./.usagePercent‚78.71720123634026¾›Ásžø‚(:“ ÙèÔꨊ¯ gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.167' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.833400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âCCvqù‚(:æ ÙèÔتŽ² disk.dm-1.out‚ 29.1953125 disk.sda.out‚ 29.1953125 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 2091.89453125! disk.nvme0n1.in‚ 2091.89453125" memory_percent‚85.115647315979( disk./.usagePercent‚78.7172037964156 disk.sda3.out‚ 29.1953125 disk.sda2.out‚0& disk.nvme0n1.out‚21772.70361328125# disk.dm-0.out‚21772.70361328125 proc.memory.rssMB‚2842.875 cpu‚24.451993401277242 network.sent‚ 1.162634e+06 network.recv‚355456% disk./.usageGB‚1172.9432220458984 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.12890625 disk.sda1.in‚0( disk.nvme0c0n1.out‚21772.70361328125 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 2091.89453125) proc.memory.availableMB‚ 2438.65234375* proc.memory.percent‚17.351531982421875 proc.cpu.threads‚17 +0ú+ _wandbruntime‚957  _runtime‚957Eºî¹Äû¾ _step‚68 & lr-AdamW/pg1‚0.0005891031061364059 & lr-AdamW/pg2‚0.0005891031061364059 trainer/global_step‚899 % _timestamp‚1.7884888579431798e+09 _runtime‚ 957.5836469420S, Èü % _timestamp‚1.7884888579431798e+09 _runtime‚ 957.583646942 _step‚68 & lr-AdamW/pg1‚0.0005891031061364059 & lr-AdamW/pg2‚0.0005891031061364059 trainer/global_step‚899Du:5Ã0ý+ _wandbruntime‚957  _runtime‚957>ü­Ï'þÉO YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 904264920576 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68777091915776 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225140736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61856121854976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34810559 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 904264920576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15195672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4823264 % _timestamp‚1.7884888579445672e+09 _step‚69 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67832017161216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 901774429712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60951067053568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13512301 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60380108897504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 901003788848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15195670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚580 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 201200640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19593369 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60951067053568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67831993221120 _runtime‚ 957.585360511 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34789039 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61855331974144 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60379321217280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 945074754560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13512881 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 944873553920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59478334467792 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61855331974144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 905036798464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8689617 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19614887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19593369 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8689615 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚578 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61856121854976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34810559 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 904264920576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15195672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4823264 % _timestamp‚1.7884888579445672e+09 _step‚69 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67832017161216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 901774429712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60951067053568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13512301 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60380108897504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 901003788848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15195670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚580 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 201200640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19593369 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60951067053568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67831993221120 _runtime‚ 957.585360511 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34789039 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61855331974144 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60379321217280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 945074754560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13512881 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 944873553920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59478334467792 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61855331974144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 905036798464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8689617 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19614887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19593369 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8689615 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚578 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15212544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 905269666304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13527297 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67907341563904 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8699265 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61924036659200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13527856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚² Ú557 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 946122141696 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34849170 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61018766992896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61018766992896 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 212544512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 trainer/global_step‚899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15212546 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59544400903552 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 906041557504 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34849170 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34827624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚559 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61924826553344 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0§zËíÜ&‚ÖM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 trainer/global_step‚899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15212546 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59544400903552 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 906041557504 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34849170 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34827624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚26617 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚559 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61924826553344 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4828034 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 945933537280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15212544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 905269666304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13527297 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67907341563904 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8699265 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61924036659200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13527856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚557 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 946122141696 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34849170 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61018766992896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61018766992896 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 212544512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840FgiÙ0ƒ+ _wandbruntime‚958  _runtime‚958Cl¬ûX„Ò /loss/FRt3/assignment_loss‚3.380985134414264 .loss/FRt2/detection_loss‚2.732346852620443 /loss/FRt4/detection_loss‚1.7841493143485143 1loss/SRqqt1/detection_loss‚2.5808166895593914 /loss/FBt1/detection_loss‚2.3947372640434064 0loss/FRt1/assignment_loss‚5.4684697860165645 2loss/SRqqt2/assignment_loss‚2.8470882585174158 .loss/FBt3/detection_loss‚4.196003754933675 2loss/SRqqt1/assignment_loss‚2.6159176434789386 /loss/FBt3/assignment_loss‚0.649831622838974 1loss/FBt4/assignment_loss‚0.30388798215548474 1loss/SRqqt4/detection_loss‚1.2349274909123777 /loss/FBt4/detection_loss‚0.8327265074478991 &loss/total_loss‚55.928753639088455 epoch‚0 _runtime‚ 958.741913229 0loss/FRt4/assignment_loss‚1.8969213240421736 1loss/SRqqt2/detection_loss‚2.4932831870882133 % _timestamp‚1.7884888591015253e+09 /loss/FRt2/assignment_loss‚5.532144066059228 /loss/FRt3/detection_loss‚2.0087050359163965 2loss/SRqqt3/assignment_loss‚1.9685540415238643 1loss/FBt2/assignment_loss‚0.46625926593939343 trainer/global_step‚899 _step‚71 0loss/FBt1/assignment_loss‚0.6773250208168878 .loss/FRt1/detection_loss‚2.783817808879049 1loss/SRqqt3/detection_loss‚1.5907922013514284 /loss/FBt2/detection_loss‚4.5676727791627245 1loss/SRqqt4/assignment_loss‚0.921390607021749üC‹²\…Ö 2loss/SRqqt1/assignment_loss‚2.6159176434789386 /loss/FBt3/assignment_loss‚0.649831622838974 1loss/FBt4/assignment_loss‚0.30388798215548474 1loss/SRqqt4/detection_loss‚1.2349274909123777 /loss/FBt4/detection_loss‚0.8327265074478991 &loss/total_loss‚55.928753639088455 epoch‚0 _runtime‚ 958.741913229 0loss/FRt4/assignment_loss‚1.8969213240421736 1loss/SRqqt2/detection_loss‚2.4932831870882133 % _timestamp‚1.7884888591015253e+09 /loss/FRt2/assignment_loss‚5.532144066059228 /loss/FRt3/detection_loss‚2.0087050359163965 2loss/SRqqt3/assignment_loss‚1.9685540415238643 1loss/FBt2/assignment_loss‚0.46625926593939343 trainer/global_step‚899 _step‚71 0loss/FBt1/assignment_loss‚0.6773250208168878 .loss/FRt1/detection_loss‚2.783817808879049 1loss/SRqqt3/detection_loss‚1.5907922013514284 /loss/FBt2/detection_loss‚4.5676727791627245 1loss/SRqqt4/assignment_loss‚0.921390607021749 /loss/FRt3/assignment_loss‚3.380985134414264 .loss/FRt2/detection_loss‚2.732346852620443 /loss/FRt4/detection_loss‚1.7841493143485143 1loss/SRqqt1/detection_loss‚2.5808166895593914 /loss/FBt1/detection_loss‚2.3947372640434064 0loss/FRt1/assignment_loss‚5.4684697860165645 2loss/SRqqt2/assignment_loss‚2.8470882585174158 .loss/FBt3/detection_loss‚4.196003754933675G-¥V“†‚(:ˆ œÙèÔÛß­ gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®kDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71599410360320 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 955288498688 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 951074373956 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚63732713590388 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64336064019968 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 954516620800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _runtime‚1008.968279815 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62781656255792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71599434300416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64336064019968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20703789 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206266368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚73 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36743161 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16039372 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16039370 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64336082022912 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 997936477184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65290580640768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14266183 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65291370521600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9172015 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72597164511232 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 997754150912 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚63733501270612 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 955288498688 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65290580640768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16039372 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5094166 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5093619 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 $ _timestamp‚1.788488909327589e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36721641 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14265634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71599410360320 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 955288498688 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 951074373956 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚63732713590388 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64336064019968 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 954516620800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _runtime‚1008.968279815 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62781656255792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71599434300416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64336064019968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20703789 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206266368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚73 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36743161 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16039372 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16039370 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64336082022912 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 997936477184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65290580640768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14266183 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65291370521600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9172015 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72597164511232 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 997754150912 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚63733501270612 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 955288498688 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65290580640768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16039372 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5094166 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5093619 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 $ _timestamp‚1.788488909327589e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36721641 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14265634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14280723 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16056244 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65359285325824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14281274 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64403781962240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64403763959296 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 955521366528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72673790954496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16056244 _step‚74 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20703982 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64403781962240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚63800568846458 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 956293257728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16056246 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 trainer/global_step‚949 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36760226 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65360075219968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 _runtime‚1009.893190642 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36781772 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16056246 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20725526 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5099060 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72673582604288 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62847722691552 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 955521366528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20725526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9181663 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71674758703104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184410112 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 998823901184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208350208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71674782643200 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 956293257728 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20703982 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9181665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 952846154906 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 952075513834 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚03£$–ØM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208350208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71674782643200 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 956293257728 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20703982 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9181665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 952846154906 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 952075513834 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14280723 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16056244 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65359285325824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14281274 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64403781962240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64403763959296 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 955521366528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72673790954496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16056244 _step‚74 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20703982 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64403781962240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚63800568846458 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 956293257728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16056246 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 trainer/global_step‚949 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36760226 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65360075219968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 _runtime‚1009.893190642 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36781772 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16056246 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20725526 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5099060 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72673582604288 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62847722691552 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 955521366528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20725526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9181663 QCDeviceStatsMonit„ù†PÛor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71674758703104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184410112 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 998823901184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664J_ùí©2—- _wandbruntime‚1009  _runtime‚1009>¦2N˜È 1loss/SRqqt3/detection_loss‚1.6299458942983462 1loss/SRqqt4/detection_loss‚1.1845088157865962 /loss/FRt3/assignment_loss‚4.001082078406685 1loss/FBt4/assignment_loss‚0.27147635835901424 /loss/FBt1/detection_loss‚2.4543504416942596 ,loss/FBt2/detection_loss‚4.0388714160238 _runtime‚1009.894778284 /loss/FRt1/assignment_loss‚4.668616959542939 /loss/FBt1/assignment_loss‚0.595392051541629 .loss/FRt2/detection_loss‚3.033036851457187 % _timestamp‚1.7884889102541325e+09 0loss/FBt2/assignment_loss‚0.6888090286935835 0loss/FBt3/assignment_loss‚0.5552875995635986 1loss/SRqqt2/detection_loss‚2.2669809601373143 .loss/FBt4/detection_loss‚0.813014450613615 0loss/FRt4/assignment_loss‚1.7102907820181412 %loss/total_loss‚55.80178194063724 _step‚75 /loss/FRt2/assignment_loss‚5.411796612398965 1loss/SRqqt1/assignment_loss‚2.906170041330399 0loss/SRqqt1/detection_loss‚2.483078263459667 trainer/global_step‚949 2loss/SRqqt2/assignment_loss‚2.5576342162158756 2loss/SRqqt3/assignment_loss‚1.8530470114471256 /loss/FRt4/detection_loss‚1.8359185419299386 .loss/FRt3/detection_loss‚2.150612723409084 .loss/FBt3/detection_loss‚4.584993263085683 epoch‚0 0loss/SRqqt4/assignment_loss‚1.16903335430612 .loss/FRt1/detection_loss‚2.937834224917672ú“–=R™Ì 0loss/SRqqt4/assignment_loss‚1.16903335430612 .loss/FRt1/detection_loss‚2.937834224917672 1loss/SRqqt3/detection_loss‚1.6299458942983462 1loss/SRqqt4/detection_loss‚1.1845088157865962 /loss/FRt3/assignment_loss‚4.001082078406685 1loss/FBt4/assignment_loss‚0.27147635835901424 /loss/FBt1/detection_loss‚2.4543504416942596 ,loss/FBt2/detection_loss‚4.0388714160238 _runtime‚1009.894778284 /loss/FRt1/assignment_loss‚4.668616959542939 /loss/FBt1/assignment_loss‚0.595392051541629 .loss/FRt2/detection_loss‚3.033036851457187 % _timestamp‚1.7884889102541325e+09 0loss/FBt2/assignment_loss‚0.6888090286935835 0loss/FBt3/assignment_loss‚0.5552875995635986 1loss/SRqqt2/detection_loss‚2.2669809601373143 .loss/FBt4/detection_loss‚0.813014450613615 0loss/FRt4/assignment_loss‚1.7102907820181412 %loss/total_loss‚55.80178194063724 _step‚75 /loss/FRt2/assignment_loss‚5.411796612398965 1loss/SRqqt1/assignment_loss‚2.906170041330399 0loss/SRqqt1/detection_loss‚2.483078263459667 trainer/global_step‚949 2loss/SRqqt2/assignment_loss‚2.5576342162158756 2loss/SRqqt3/assignment_loss‚1.8530470114471256 /loss/FRt4/detection_loss‚1.8359185419299386 .loss/FRt3/detection_loss‚2.150612723409084 .loss/FBt3/detection_loss‚4.584993263085683 epoch‚0K4Ykæ“š‚(:ˆ ØÙèԃ䖮 gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f,Bûx›‚(:í ØÙèÔúû„³) proc.memory.availableMB‚ 2438.26171875* proc.memory.percent‚17.351555824279785 proc.cpu.threads‚17 network.recv‚376271! disk.nvme0n1.in‚ 2220.03515625& disk.nvme0n1.out‚23444.97216796875 disk.dm-0.in‚ 2220.03515625 disk.sda3.out‚ 31.4765625 disk.sda1.out‚0 network.sent‚ 1.233562e+06) disk./.usagePercent‚78.75075716788326 disk.sda2.out‚0( disk.nvme0c0n1.out‚23444.97216796875 disk.dm-1.in‚0 disk.dm-1.out‚ 31.4765625 disk.drbd1260.out‚ 0.13671875 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 31.4765625 disk.sda1.in‚0$ memory_percent‚85.11803150177002# proc.memory.rssMB‚ 2842.87890625 cpu‚24.49726175871358% disk./.usageGB‚1173.4431915283203 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 2220.03515625# disk.dm-0.out‚23444.97216796875,¡®÷žœ‚(:“ çÙèÔØÄ°¯ gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.562599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cß x‚(:í çÙèÔ¥›£³ network.recv‚380897( disk.nvme0c0n1.out‚23859.35595703125# disk.dm-0.out‚23859.35595703125 disk.sda3.in‚0 disk.dm-1.out‚ 31.77734375) proc.memory.availableMB‚ 2438.75390625( disk./.usagePercent‚78.7507558878456 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 2220.03515625 disk.dm-0.in‚ 2220.03515625 disk.dm-1.in‚0 disk.sda.out‚ 31.77734375$ memory_percent‚85.11502742767334 cpu‚24.505582042300997 network.sent‚ 1.282392e+06 disk.sda2.out‚0! disk.nvme0n1.in‚ 2220.03515625& disk.nvme0n1.out‚23859.35595703125 disk.sda3.out‚ 31.77734375 disk.drbd1260.in‚0 disk.drbd1260.out‚0.140625 disk.sda.in‚0$ disk./.usageGB‚1173.443172454834 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2842.87890625* proc.memory.percent‚17.351555824279785 proc.cpu.threads‚17Â[”ž‚(:’ öÙèÔ«ö® gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.96' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.992000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£qNUvŸ‚(:ë öÙèÔ­¾î° proc.cpu.threads‚17 disk.sda.out‚ 32.16796875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0$ memory_percent‚85.11500358581543# proc.memory.rssMB‚ 2842.87890625( disk./.usagePercent‚78.7507558878456 disk.drbd1260.out‚0.140625 disk.sda1.in‚0& disk.nvme0n1.out‚24250.15283203125 disk.sda3.in‚0* proc.memory.percent‚17.351555824279785 network.sent‚ 1.287354e+06 disk.drbd1260.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 2220.03515625! disk.nvme0n1.in‚ 2220.03515625 disk.dm-0.in‚ 2220.03515625# disk.dm-0.out‚24250.15283203125 disk.sda3.out‚ 32.16796875 network.recv‚384187$ disk./.usageGB‚1173.443172454834 disk.sda.in‚0( disk.nvme0c0n1.out‚24250.15283203125 disk.dm-1.out‚ 32.16796875( proc.memory.availableMB‚ 2438.7578125 cpu‚24.51357813656157¢¦Qò2 - _wandbruntime‚1061  _runtime‚1061 Ñá½ _runtime‚ 1061.1302088 _step‚76 & lr-AdamW/pg1‚0.0006546318164964065 & lr-AdamW/pg2‚0.0006546318164964065 trainer/global_step‚999 % _timestamp‚1.7884889614898863e+09Å«»žÇ¢Á & lr-AdamW/pg1‚0.0006546318164964065 & lr-AdamW/pg2‚0.0006546318164964065 trainer/global_step‚999 % _timestamp‚1.7884889614898863e+09 _runtime‚ 1061.1302088 _step‚76LÖÑȸ2£- _wandbruntime‚1061  _runtime‚1061Y¹k¢Ø'¤ÒO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21792691 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1005540198912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1004768321024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16883072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194909184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9654415 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15019163 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 trainer/global_step‚999 _runtime‚1061.131403762 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚67721060986368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚575 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1004768321024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16883072 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76417401354240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1050549914624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1001144959064 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68726619188224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38675763 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1005540198912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15018588 % _timestamp‚1.7884889614909284e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16883070 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21771173 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5364173 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68725829307392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5364746 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚67086893643720 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚67086105963496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚67721060986368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1050355005440 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68725829307392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚67721078989312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15019163 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 trainer/global_step‚999 _runtime‚1061.131403762 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚67721060986368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚575 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1004768321024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16883072 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76417401354240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1050549914624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1001144959064 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68726619188224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38675763 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1005540198912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15018588 % _timestamp‚1.7884889614909284e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16883070 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21771173 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5364173 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68725829307392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5364746 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚67086893643720 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚67086105963496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚67721060986368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1050355005440 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68725829307392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚67721078989312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76493586243584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚67153173539134 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 _runtime‚1062.003304039 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38692828 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21814428 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76493800885248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38714374 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9664065 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚66151044479552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1002146098942 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚67788760925696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5370119 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1002916740014 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38714374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15033632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 trainer/global_step‚999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38692828 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚67788778928640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1005773066752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_byåëWýtes.small_pool.freed‚ 1051410401280 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5369569 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚67788760925696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 214641664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16899944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15034184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚78 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21814428 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68795323886592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1006544957952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16899944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚67788778928640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1006544957952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190701568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚67153961219566 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68794533992448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 % _timestamp‚1.7884889623623052e+09 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16899944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚67788778928640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1006544957952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190701568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚67153961219566 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68794533992448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 % _timestamp‚1.7884889623623052e+09 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76493586243584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚67153173539134 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 _runtime‚1062.003304039 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38692828 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21814428 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76493800885248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38714374 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9664065 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚66151044479552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1002146098942 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚67788760925696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5370119 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1002916740014 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38714374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15033632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 trainer/global_step‚999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38692828 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚67788778928640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1005773066752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1051410401280 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5369569 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚67788760925696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 214641664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16899944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15034184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚78 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21814428 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68795323886592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12152995840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004NßøNê2©- _wandbruntime‚1062  _runtime‚1062qïïSªÍ /loss/FRt2/assignment_loss‚5.136715999016395 1loss/FBt4/assignment_loss‚0.29371409434962675 .loss/FRt1/detection_loss‚2.647239477206499 0loss/SRqqt2/detection_loss‚2.691622762768357 1loss/SRqqt3/detection_loss‚1.5809439026134116 /loss/FRt1/assignment_loss‚4.376114246172783 2loss/SRqqt3/assignment_loss‚2.1759085114782963 2loss/FBt3/assignment_loss‚0.033887348137795925 /loss/FRt2/detection_loss‚3.8439840445151696 /loss/FBt3/detection_loss‚3.7950194478034973 /loss/FBt4/detection_loss‚0.6563519854849066 %loss/total_loss‚53.15360876193292 $ _timestamp‚1.788488962364172e+09 2loss/SRqqt1/assignment_loss‚2.5389007654573237 /loss/FRt3/detection_loss‚2.0793793714708753 _step‚79 1loss/SRqqt2/assignment_loss‚2.122986391738609 1loss/SRqqt4/assignment_loss‚1.301618588892253 0loss/FRt4/assignment_loss‚2.0852928532744355 /loss/FRt4/detection_loss‚1.6863096182076436 0loss/SRqqt1/detection_loss‚2.510547629424504 _runtime‚ 1062.00433945 0loss/FBt1/assignment_loss‚0.8147291819543832 -loss/FBt2/assignment_loss‚0.6673268658871 trainer/global_step‚999 /loss/FRt3/assignment_loss‚3.853248220902902 1loss/SRqqt4/detection_loss‚1.0853661637949317 /loss/FBt1/detection_loss‚2.4838233332742345 /loss/FBt2/detection_loss‚2.6925779581069946 epoch‚0è#ÎW«Ñ 0loss/FBt1/assignment_loss‚0.8147291819543832 -loss/FBt2/assignment_loss‚0.6673268658871 trainer/global_step‚999 /loss/FRt3/assignment_loss‚3.853248220902902 1loss/SRqqt4/detection_loss‚1.0853661637949317 /loss/FBt1/detection_loss‚2.4838233332742345 /loss/FBt2/detection_loss‚2.6925779581069946 epoch‚0 /loss/FRt2/assignment_loss‚5.136715999016395 1loss/FBt4/assignment_loss‚0.29371409434962675 .loss/FRt1/detection_loss‚2.647239477206499 0loss/SRqqt2/detection_loss‚2.691622762768357 1loss/SRqqt3/detection_loss‚1.5809439026134116 /loss/FRt1/assignment_loss‚4.376114246172783 2loss/SRqqt3/assignment_loss‚2.1759085114782963 2loss/FBt3/assignment_loss‚0.033887348137795925 /loss/FRt2/detection_loss‚3.8439840445151696 /loss/FBt3/detection_loss‚3.7950194478034973 /loss/FBt4/detection_loss‚0.6563519854849066 %loss/total_loss‚53.15360876193292 $ _timestamp‚1.788488962364172e+09 2loss/SRqqt1/assignment_loss‚2.5389007654573237 /loss/FRt3/detection_loss‚2.0793793714708753 _step‚79 1loss/SRqqt2/assignment_loss‚2.122986391738609 1loss/SRqqt4/assignment_loss‚1.301618588892253 0loss/FRt4/assignment_loss‚2.0852928532744355 /loss/FRt4/detection_loss‚1.6863096182076436 0loss/SRqqt1/detection_loss‚2.510547629424504 _runtime‚ 1062.00433945OHü½“¬‚(:ˆ …ÚèÔºòù® gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9564 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œüIµs­‚(:è …ÚèÔš§„± cpu‚24.521251266760977 proc.cpu.threads‚17$ disk./.usageGB‚1173.442211151123 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.14453125" proc.memory.rssMB‚ 2842.8828125 network.sent‚ 1.293664e+06) disk./.usagePercent‚78.75069137394696 disk.sda.out‚ 32.64453125 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 2220.03515625! disk.nvme0n1.in‚ 2220.03515625% disk.nvme0n1.out‚24689.2373046875 disk.dm-0.in‚ 2220.03515625 disk.sda3.in‚0 disk.sda3.out‚ 32.64453125 disk.dm-1.out‚ 32.64453125 disk.drbd1260.in‚0 disk.sda.in‚0' disk.nvme0c0n1.out‚24689.2373046875# memory_percent‚85.1186752319336* proc.memory.percent‚17.351579666137695 network.recv‚389024 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0" disk.dm-0.out‚24689.2373046875& proc.memory.availableMB‚ 2438.156252§Â:ž®‚(:“ ”ÚèÔÜ»ý® gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.175' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.634999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0??æíg¯‚(:Ü ”ÚèÔ®ýȲ" proc.memory.rssMB‚ 2842.8828125* proc.memory.percent‚17.351579666137695 cpu‚24.52888190382342% disk./.usageGB‚1173.4422302246094 disk.nvme0c0n1.in‚ 2353.46875 disk.nvme0n1.in‚ 2353.46875 disk.dm-1.out‚ 33.2734375 disk.sda.in‚0 disk.sda.out‚ 33.2734375$ memory_percent‚85.11543273925781% proc.memory.availableMB‚ 2438.6875 network.recv‚394420 disk.sda2.out‚0" disk.dm-0.out‚25104.1123046875 disk.sda3.in‚0 disk.sda3.out‚ 33.2734375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.14453125 disk.sda1.out‚0 network.sent‚ 1.343332e+06 disk.dm-0.in‚ 2353.46875 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.75069265398464 disk.sda2.in‚0' disk.nvme0c0n1.out‚25104.1123046875% disk.nvme0n1.out‚25104.1123046875 disk.dm-1.in‚0$¸B“°‚(:ˆ £ÚèÔê¤ã¯ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.4658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P©P]±‚(:Ò £ÚèÔÕÇÓ° disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.15234375( proc.memory.availableMB‚ 2438.9296875 disk.nvme0n1.in‚ 2353.46875% disk./.usageGB‚1173.4422149658203 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 cpu‚24.53667945321827" disk.dm-0.out‚25423.0751953125 disk.sda3.out‚33.625 disk.sda.out‚33.625" proc.memory.rssMB‚ 2842.8828125 proc.cpu.threads‚17% disk.nvme0n1.out‚25423.0751953125 disk.dm-0.in‚ 2353.46875 network.sent‚ 1.350423e+06 network.recv‚398171( disk./.usagePercent‚78.7506916299545 disk.nvme0c0n1.in‚ 2353.46875$ memory_percent‚85.11395454406738* proc.memory.percent‚17.351579666137695 disk.sda1.out‚0' disk.nvme0c0n1.out‚25423.0751953125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚33.625HDéú“²‚(:ˆ ²ÚèÔ’ˆÚ® gpu.0.gpu‚13.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.759' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.3518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v´äm³‚(:â ²ÚèÔ±êͲ disk.sda3.out‚ 34.08984375 disk.dm-1.in‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚25598.5205078125* proc.memory.percent‚17.351579666137695 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 2353.46875" proc.memory.rssMB‚ 2842.8828125 cpu‚24.544116004909938 disk.sda.out‚ 34.08984375 network.sent‚ 1.353585e+06) disk./.usagePercent‚78.75070980648944 disk.drbd1260.out‚ 0.15234375 disk.sda.in‚0 disk.dm-0.in‚ 2353.46875" disk.dm-0.out‚25589.2705078125( proc.memory.availableMB‚ 2438.3828125 network.recv‚401742% disk./.usageGB‚1173.4424858093262 disk.dm-1.out‚ 34.08984375 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 2353.46875' disk.nvme0c0n1.out‚25598.0205078125# memory_percent‚85.1172924041748 disk.sda3.in‚0¨ö±Q2´- _wandbruntime‚1114  _runtime‚1114üúÌÂĵ¾ trainer/global_step‚1049 # _timestamp‚1.78848901465466e+09 _runtime‚1114.295060496 _step‚80 & lr-AdamW/pg1‚0.0006873961716764068 & lr-AdamW/pg2‚0.0006873961716764068€w-<ȶ _runtime‚1114.295060496 _step‚80 & lr-AdamW/pg1‚0.0006873961716764068 & lr-AdamW/pg2‚0.0006873961716764068 trainer/global_step‚1049 # _timestamp‚1.78848901465466e+09PÜ(2·- _wandbruntime‚1114  _runtime‚11149“èÀ!¸ÕO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1103218963456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 trainer/global_step‚1049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1055791899136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚69388282792432 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80237678257152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70440286016828 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40586845 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70439498336604 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40608365 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10136815 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1055020021248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17726772 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214654976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17726772 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69388299831792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22860075 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15772759 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17726770 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10136817 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1103409678336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72161867854848 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72161077974016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79134268578816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190714880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22881593 % _timestamp‚1.7884890146557899e+09 _step‚81 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚71106075955712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 _runtime‚1114.296313518 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1055020021248 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72161077974016 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚71106075955712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5635400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1051215544172 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚71106057952768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17726770 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488RIž¢ß'¹ÙO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22881593 % _timestamp‚1.7884890146557899e+09 _step‚81 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚71106075955712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 _runtime‚1114.296313518 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1055020021248 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72161077974016 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚71106075955712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5635400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1051215544172 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚71106057952768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17726770 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1103218963456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 trainer/global_step‚1049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1055791899136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚69388282792432 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80237678257152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70440286016828 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40586845 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70439498336604 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40608365 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10136815 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1055020021248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17726772 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 214654976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17726772 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69388299831792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22860075 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15772759 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17726770 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10136817 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1103409678336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72161867854848 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72161077974016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79134268578816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190714880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004QŽ~w2º- _wandbruntime‚1115  _runtime‚1115^­ŒTå&»ßM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40625430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22903330 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5641353 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188604416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22881786 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17743644 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1056796658176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1056024766976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79209616921600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1056024766976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17743644 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15787818 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 _step‚82 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1104281576448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚71173757892096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚72229782659072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 212544512 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72230572553216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1052987325122 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1115.146299975 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚71173757892096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22903330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40646976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70506565912242 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70507353592674 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10146463 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80314087102464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69454349228192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚71173775895040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22881786 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10146465 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15787278 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1056796658176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79209592981504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840ÅÏzoé&¼ãM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1115.146299975 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚71173757892096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22903330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40646976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70506565912242 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70507353592674 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10146463 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80314087102464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69454349228192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚71173775895040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22881786 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10146465 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15787278 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1056796658176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79209592981504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12152995840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40625430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22903330 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5641353 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188604416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22881786 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17743644 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1056796658176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1056024766976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79209616921600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1056024766976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 771891200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17743644 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15787818 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 _step‚82 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1104281576448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚71173757892096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚72229782659072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 212544512 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72230572553216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1052987325122 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1106043599360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42540967 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18570472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚75596326640640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23970495 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5907892 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10619217 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73792890709712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192812032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72691604580432 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18570470 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23948977 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74491072922112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5908443 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1106043599360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10619215 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74491054919168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84057895601152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1105271721472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16527107 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 _runtime‚1166.781144803 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1102056770144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1105271721472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚10751ΰYß'ËÙO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18570472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚75596326640640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23970495 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5907892 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10619217 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73792890709712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192812032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1023410176 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72691604580432 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18570470 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23948977 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74491072922112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5908443 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1106043599360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10619215 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74491054919168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84057895601152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1105271721472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16527107 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 _runtime‚1166.781144803 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1102056770144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1105271721472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1106043599360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42540967 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74558754858496 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74558772861440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚75665821219840 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 trainer/global_step‚1099 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚73859958285350 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10628863 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1106276467200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23992232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23970688 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚74558772861440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74558754858496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1157476307968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42579578 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23970688 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84134295727104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1107048358400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1107048358400 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5913343 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚562 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82977010120704 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42579578 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16542206 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚84134510368768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18587346 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚73860745965782 _runtime‚1167.615238223 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1157285606400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2ಙzé&ÎãM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚75665821219840 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 trainer/global_step‚1099 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚73859958285350 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10628863 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1106276467200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23992232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23970688 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚74558772861440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74558754858496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1157476307968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42579578 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23970688 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84134295727104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1107048358400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1107048358400 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5913343 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚562 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82977010120704 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42579578 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16542206 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚84134510368768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18587346 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚73860745965782 _runtime‚1167.615238223 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1157285606400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18587344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1106276467200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74558754858496 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74558772861440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072VÜù×2Ï- _wandbruntime‚1167  _runtime‚1167uƒV¬TÐÎ 2loss/SRqqt3/assignment_loss‚1.9710845356950393 %loss/total_loss‚51.57741804566801 0loss/FRt4/assignment_loss‚1.2100958990496258 1loss/FBt3/assignment_loss‚0.04491952042250583 1loss/SRqqt2/detection_loss‚2.6399726884232626 1loss/SRqqt1/detection_loss‚2.5919733366795947 .loss/FBt1/detection_loss‚2.731760573387146 _step‚87 epoch‚0 trainer/global_step‚1099 2loss/SRqqt4/assignment_loss‚1.0350952975097156 0loss/FBt4/assignment_loss‚0.2459860653802094 1loss/SRqqt4/detection_loss‚1.2185529259935257 /loss/FBt4/detection_loss‚0.7085161303575348 _runtime‚ 1167.61727924 /loss/FBt3/detection_loss‚4.6288126309712725 2loss/SRqqt2/assignment_loss‚2.1228271854213543 0loss/FBt2/assignment_loss‚0.5050701828939574 /loss/FRt2/detection_loss‚2.5046874403953554 .loss/FRt3/detection_loss‚2.132228205512677 0loss/SRqqt3/detection_loss‚1.635580606185473 /loss/FRt1/assignment_loss‚4.431907271345456 .loss/FRt4/detection_loss‚1.890138831249503 .loss/FBt2/detection_loss‚3.197686893599374 $ _timestamp‚1.788489067977114e+09 /loss/FRt3/assignment_loss‚3.909909238772733 0loss/FBt1/assignment_loss‚0.5925949098542332 .loss/FRt1/detection_loss‚2.528202708810568 /loss/FRt2/assignment_loss‚4.757112836837768 2loss/SRqqt1/assignment_loss‚2.3427021309201206å•XÑÒ 1loss/SRqqt4/detection_loss‚1.2185529259935257 /loss/FBt4/detection_loss‚0.7085161303575348 _runtime‚ 1167.61727924 /loss/FBt3/detection_loss‚4.6288126309712725 2loss/SRqqt2/assignment_loss‚2.1228271854213543 0loss/FBt2/assignment_loss‚0.5050701828939574 /loss/FRt2/detection_loss‚2.5046874403953554 .loss/FRt3/detection_loss‚2.132228205512677 0loss/SRqqt3/detection_loss‚1.635580606185473 /loss/FRt1/assignment_loss‚4.431907271345456 .loss/FRt4/detection_loss‚1.890138831249503 .loss/FBt2/detection_loss‚3.197686893599374 $ _timestamp‚1.788489067977114e+09 /loss/FRt3/assignment_loss‚3.909909238772733 0loss/FBt1/assignment_loss‚0.5925949098542332 .loss/FRt1/detection_loss‚2.528202708810568 /loss/FRt2/assignment_loss‚4.757112836837768 2loss/SRqqt1/assignment_loss‚2.3427021309201206 2loss/SRqqt3/assignment_loss‚1.9710845356950393 %loss/total_loss‚51.57741804566801 0loss/FRt4/assignment_loss‚1.2100958990496258 1loss/FBt3/assignment_loss‚0.04491952042250583 1loss/SRqqt2/detection_loss‚2.6399726884232626 1loss/SRqqt1/detection_loss‚2.5919733366795947 .loss/FBt1/detection_loss‚2.731760573387146 _step‚87 epoch‚0 trainer/global_step‚1099 2loss/SRqqt4/assignment_loss‚1.0350952975097156 0loss/FBt4/assignment_loss‚0.2459860653802094W=…ا“Ò‚(:ˆ îÚèÔô¡¬® gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I€1öqÓ‚(:æ îÚèÔàŒ¯ network.sent‚ 1.419632e+06) disk./.usagePercent‚78.56541001716111 disk.dm-1.out‚ 35.7109375 disk.drbd1260.out‚ 0.16015625 disk.sda1.in‚0$ memory_percent‚85.11936664581299) proc.memory.availableMB‚ 2438.04296875" proc.memory.rssMB‚ 2842.8828125 network.recv‚418875 disk.sda3.out‚ 35.7109375 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚27216.4072265625 disk.dm-0.in‚ 2533.6640625% disk./.usageGB‚1170.6813850402832 disk.dm-1.in‚0 disk.sda.out‚ 35.7109375" disk.nvme0c0n1.in‚ 2533.6640625% disk.nvme0n1.out‚27216.4072265625" disk.dm-0.out‚27216.4072265625 cpu‚24.57053534011288 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 2533.6640625* proc.memory.percent‚17.351579666137695 proc.cpu.threads‚17sñ~¶“Ô‚(:ˆ ýÚèÔÛ¥€­ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.526' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.1052 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=¡)vmÕ‚(:â ýÚèÔۧȰ disk.dm-0.in‚ 2533.6640625" proc.memory.rssMB‚ 2842.8828125 proc.cpu.threads‚17) disk./.usagePercent‚78.57615798147111$ disk./.usageGB‚1170.841537475586 disk.sda3.out‚36.15625 disk.sda.in‚0 disk.nvme0n1.in‚ 2533.6640625& disk.nvme0n1.out‚27523.60400390625# disk.dm-0.out‚27523.60400390625$ memory_percent‚85.11793613433838 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.out‚36.15625 disk.sda1.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚27523.60400390625) proc.memory.availableMB‚ 2438.27734375* proc.memory.percent‚17.351579666137695 network.sent‚ 1.469878e+06 disk.dm-1.out‚36.15625 disk.drbd1260.out‚ 0.1640625 disk.sda1.out‚0 cpu‚24.576556114186477 network.recv‚424887 disk.drbd1260.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 2533.6640625§²¹OžÖ‚(:“ ŒÛèÔ—‡†® gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.554' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.110800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-øTybׂ(:× ŒÛèÔö†§¯$ memory_percent‚85.11784076690674" proc.memory.rssMB‚ 2842.8828125* proc.memory.percent‚17.351579666137695 disk.dm-0.out‚ 27620.0390625 disk.sda3.out‚36.5625) proc.memory.availableMB‚ 2438.29296875 cpu‚24.582544678484478 proc.cpu.threads‚17 network.sent‚ 1.475823e+06 network.recv‚429036% disk./.usageGB‚1170.7482795715332 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚36.5625 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚78.56989936528154 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 2726.80078125" disk.nvme0n1.out‚ 27620.0390625 disk.dm-0.in‚ 2726.80078125 disk.sda3.in‚0 disk.dm-1.out‚36.5625 disk.drbd1260.out‚ 0.1640625 disk.sda2.out‚0$ disk.nvme0c0n1.out‚ 27620.0390625! disk.nvme0n1.in‚ 2726.80078125 disk.dm-1.in‚0¬ûù~2Ø- _wandbruntime‚1214  _runtime‚1214ý€;¬ÅÙ¿ & lr-AdamW/pg1‚0.0007529248820364074 & lr-AdamW/pg2‚0.0007529248820364074 trainer/global_step‚1149 $ _timestamp‚1.788489114876041e+09 _runtime‚1214.516427858 _step‚88L?…ÉÚà & lr-AdamW/pg1‚0.0007529248820364074 & lr-AdamW/pg2‚0.0007529248820364074 trainer/global_step‚1149 $ _timestamp‚1.788489114876041e+09 _runtime‚1214.516427858 _step‚88XØŒ`42Û- _wandbruntime‚1214  _runtime‚1214?Ä?Û'ÜÕO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19414172 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75994926368432 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1152127355252 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86669102857216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25037879 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77146283082820 _step‚89 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77147070763044 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 trainer/global_step‚1149 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1156295299584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1155523421696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19414170 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6179589 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17281759 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44452049 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19414170 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86669078917120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1209132402176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77876051885568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186520576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1151356714388 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11101615 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87878235259392 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87878024798720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79032365188096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19414172 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79032365188096 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25059397 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79031575307264 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75994943407792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79031575307264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚77876069888512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1208945881600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1156295299584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77876069888512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17281204 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6180142 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210460672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864…Þê=«ÝÙO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12152995840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 trainer/global_step‚1149 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 789880832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1156295299584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1155523421696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19414170 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6179589 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batcâ)w,4"h_start/segment.all.allocated‚1004 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17281759 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚488 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44452049 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19414170 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86669078917120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1209132402176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77876051885568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186520576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1151356714388 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11101615 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87878235259392 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87878024798720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚488 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79032365188096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19414172 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79032365188096 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25059397 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79031575307264 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75994943407792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79031575307264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚77876069888512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1208945881600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1156295299584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77876069888512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17281204 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6180142 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210460672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19414172 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75994926368432 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1152127355252 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚26617 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86669102857216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25037879 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787680224 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77146283082820 _step‚89 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77147070763044Yµ “Þ‚(:ˆ ›ÛèÔ•Žà­ gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9564 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xc\ zß‚(:ï ›ÛèԢݫ¯ network.sent‚ 1.480948e+06# disk.dm-0.out‚28097.86474609375 disk.sda2.in‚0" proc.memory.rssMB‚ 2842.8828125* proc.memory.percent‚17.351579666137695 network.recv‚432931% disk./.usageGB‚1170.9069709777832 disk.dm-1.out‚ 36.92578125 disk.sda.in‚0 disk.sda.out‚ 36.92578125 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚85.11903285980225! disk.nvme0n1.in‚ 3933.30078125 disk.sda3.out‚ 36.92578125# disk.nvme0c0n1.in‚ 3933.30078125( disk.nvme0c0n1.out‚28097.86474609375) proc.memory.availableMB‚ 2438.09765625 cpu‚24.58814884191623 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.16796875) disk./.usagePercent‚78.58054927870593& disk.nvme0n1.out‚28097.86474609375 disk.dm-0.in‚ 3933.30078125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0¤ÿÏ¿2à- _wandbruntime‚1215  _runtime‚1215C½îÆä&áÞM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79101069886464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79101069886464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77214138338890 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17296266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196993024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76060992804192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 _runtime‚ 1215.37305006 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25059590 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220933120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1209993443840 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44512180 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25059590 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86744427259904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884891157320151e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19431044 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19431046 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11111263 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19431046 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77943769827840 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1152357854266 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17296815 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44512180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25081134 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44490634 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86744451200000 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11111265 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1153128495338 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _step‚90ß*™&è&ââM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17296266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196993024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76060992804192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 _runtime‚ 1215.37305006 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25059590 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220933120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1209993443840 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44512180 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25059590 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86744427259904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884891157320151e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19431044 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19431046 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11111263 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19431046 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77943769827840 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1152357854266 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17296815 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44512180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25081134 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44490634 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86744451200000 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11111265 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1153128495338 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _step‚90 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79101069886464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79101069886464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017627277 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77214138338890ZЈVõ2ã- _wandbruntime‚1215  _runtime‚1215”4ÂZäÔ /loss/FRt3/detection_loss‚2.2143402929399527 /loss/FRt1/assignment_loss‚3.845720209354578 1loss/SRqqt3/assignment_loss‚2.321202588627736 0loss/FBt4/assignment_loss‚0.1719315148910862 /loss/FRt2/detection_loss‚2.6578269931110174 1loss/SRqqt4/detection_loss‚1.0735274863739808 /loss/FBt3/detection_loss‚2.6742404202620187 /loss/FBt4/detection_loss‚0.6334956793324521 0loss/FRt4/assignment_loss‚1.7330490946769714 1loss/SRqqt2/detection_loss‚1.9609766770964083 .loss/FBt1/detection_loss‚4.706679582595825 % _timestamp‚1.7884891157346828e+09 _step‚91 1loss/FBt1/assignment_loss‚0.01356564598154364 1loss/FBt3/assignment_loss‚0.18452289911995953 &loss/total_loss‚49.862390652333104 epoch‚0 trainer/global_step‚1149 /loss/FRt3/assignment_loss‚4.054248800464705 1loss/SRqqt1/assignment_loss‚2.091973010355081 /loss/FRt4/detection_loss‚1.5764851228757337 1loss/SRqqt1/detection_loss‚2.1936747340055613 .loss/FBt2/detection_loss‚3.008190373579661 2loss/SRqqt4/assignment_loss‚0.8531788464663832 /loss/FRt1/detection_loss‚2.6906344946040663 1loss/SRqqt3/detection_loss‚1.5641236672798793 _runtime‚1215.374841216 /loss/FRt2/assignment_loss‚4.757104886544718 2loss/SRqqt2/assignment_loss‚2.2402660098412763 0loss/FBt2/assignment_loss‚0.6414316219525062a%PÈöåØ /loss/FRt2/assignment_loss‚4.757104886544718 2loss/SRqqt2/assignment_loss‚2.2402660098412763 0loss/FBt2/assignment_loss‚0.6414316219525062 /loss/FRt3/detection_loss‚2.2143402929399527 /loss/FRt1/assignment_loss‚3.845720209354578 1loss/SRqqt3/assignment_loss‚2.321202588627736 0loss/FBt4/assignment_loss‚0.1719315148910862 /loss/FRt2/detection_loss‚2.6578269931110174 1loss/SRqqt4/detection_loss‚1.0735274863739808 /loss/FBt3/detection_loss‚2.6742404202620187 /loss/FBt4/detection_loss‚0.6334956793324521 0loss/FRt4/assignment_loss‚1.7330490946769714 1loss/SRqqt2/detection_loss‚1.9609766770964083 .loss/FBt1/detection_loss‚4.706679582595825 % _timestamp‚1.7884891157346828e+09 _step‚91 1loss/FBt1/assignment_loss‚0.01356564598154364 1loss/FBt3/assignment_loss‚0.18452289911995953 &loss/total_loss‚49.862390652333104 epoch‚0 trainer/global_step‚1149 /loss/FRt3/assignment_loss‚4.054248800464705 1loss/SRqqt1/assignment_loss‚2.091973010355081 /loss/FRt4/detection_loss‚1.5764851228757337 1loss/SRqqt1/detection_loss‚2.1936747340055613 .loss/FBt2/detection_loss‚3.008190373579661 2loss/SRqqt4/assignment_loss‚0.8531788464663832 /loss/FRt1/detection_loss‚è6#uh2.6906344946040663 1loss/SRqqt3/detection_loss‚1.5641236672798793 _runtime‚1215.374841216[ã ‰ žæ‚(:“ ªÛèÔªÅɯ gpu.0.gpu‚24.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚37.0 gpu.0.powerWatts‚159.589' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.917800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06܈Jjç‚(:ß ªÛèÔùߦ² disk.sda2.in‚0$ memory_percent‚85.11643409729004" proc.memory.rssMB‚ 2842.8828125 cpu‚24.59370321277937 proc.cpu.threads‚17) disk./.usagePercent‚78.58052828608814% disk./.usageGB‚1170.9066581726074! disk.nvme0n1.in‚ 3933.30078125& disk.nvme0n1.out‚28442.52490234375 disk.drbd1260.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 3933.30078125 disk.sda3.in‚0 disk.sda3.out‚37.375 disk.sda.out‚37.375( proc.memory.availableMB‚ 2438.5234375* proc.memory.percent‚17.351579666137695 disk.dm-0.in‚ 3933.30078125# disk.dm-0.out‚28442.52490234375 disk.dm-1.out‚37.375 disk.drbd1260.out‚ 0.16796875 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚28442.52490234375 network.sent‚ 1.518409e+06 network.recv‚437434 disk.dm-1.in‚0Ö㋨“è‚(:ˆ ¹ÛèÔ͸ˆ­ gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚37.0 gpu.0.powerWatts‚165.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03Gúvé‚(:ë ¹ÛèÔ‚…ò®( disk.nvme0c0n1.out‚28723.19873046875# disk.dm-0.out‚28723.19873046875 disk.dm-1.out‚ 37.8984375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.17578125 disk.sda1.in‚0" proc.memory.rssMB‚ 2842.8828125! disk.nvme0n1.in‚ 3933.30078125 disk.dm-0.in‚ 3933.30078125 disk.sda.out‚ 37.8984375 disk.sda2.in‚0' proc.memory.availableMB‚ 2438.515625* proc.memory.percent‚17.351579666137695 proc.cpu.threads‚17 network.recv‚442851) disk./.usagePercent‚78.57925208852996# disk.nvme0c0n1.in‚ 3933.30078125 disk.sda3.in‚0 disk.sda3.out‚ 37.8984375 disk.sda.in‚0 disk.sda2.out‚0 cpu‚24.599564832009808% disk./.usageGB‚1170.8876419067383& disk.nvme0n1.out‚28723.19873046875 disk.dm-1.in‚0 disk.sda1.out‚0$ memory_percent‚85.11648178100586 network.sent‚ 1.557904e+06p5Aê‚(:„ ÈÛèÔ·˜ˆ¯ gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.9' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.38 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ vë‚(:ë ÈÛèÔÚßý²$ memory_percent‚85.11648178100586 network.sent‚ 1.565705e+06 disk.sda1.in‚0 disk.dm-0.in‚ 4052.70703125 disk.sda3.in‚0 disk.drbd1260.in‚0' proc.memory.availableMB‚ 2438.515625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 4052.70703125 disk.sda3.out‚ 38.4140625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.17578125 disk.sda.in‚0" proc.memory.rssMB‚ 2842.8828125* proc.memory.percent‚17.351579666137695% disk./.usageGB‚1170.9112014770508 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 4052.70703125 cpu‚24.605011250154263 proc.cpu.threads‚17 network.recv‚447366) disk./.usagePercent‚78.58083319106142( disk.nvme0c0n1.out‚28879.67919921875& disk.nvme0n1.out‚28879.67919921875# disk.dm-0.out‚28879.67919921875 disk.dm-1.out‚ 38.4140625 disk.sda.out‚ 38.4140625š]2ì- _wandbruntime‚1262  _runtime‚1262„4ÆíÀ & lr-AdamW/pg1‚0.0007856892372164079 & lr-AdamW/pg2‚0.0007856892372164079 trainer/global_step‚1199 % _timestamp‚1.7884891624364796e+09 _runtime‚1262.077068811 _step‚92’e†EÊîÄ _step‚92 & lr-AdamW/pg1‚0.0007856892372164079 & lr-AdamW/pg2‚0.0007856892372164079 trainer/global_step‚1199 % _timestamp‚1.7884891624364796e+09 _runtime‚1262.077068811\îpÄ]2ï- _wandbruntime‚1262  _runtime‚1262Èâ–<Û'ðÕO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1206546999808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 _runtime‚1262.078350136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1205775121920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79298265195792 % _timestamp‚1.7884891624378006e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46406171 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18035073 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79298248156432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11584017 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1261769020416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81261066854912 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82466823973888 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26148299 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216752128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90436496056320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1261961832448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82467613854720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20257870 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18035642 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26126781 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192812032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1201427299496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚567 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20257870 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26148299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚569 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80499675455928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20257872 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82467613854720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91698481828864 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91698265076736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _step‚93 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82466823973888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1205775121920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81261048851968 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81261048851968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20257872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11584015 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6451058 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90436519996416 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80500463136152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1202197940360@5÷ß'ñÙO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 789880832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91698481828864 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91698265076736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12152995840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _step‚93 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82466823973888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1205775121920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81261048851968 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81261048851968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1018907648 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12152995840 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚21520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20257872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11584015 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11891237376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6451058 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90436519996416 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80500463136152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1202197940360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 771877888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1206546999808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚488 _runtime‚1262.078350136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1205775121920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚26348 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79298265195792 % _timestamp‚1.7884891624378006e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46406171 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18035073 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79298248156432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017627277 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚21518 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11584017 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1261769020416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81261066854912 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82466823973888 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26148299 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 216752128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90436496056320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1004 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1261961832448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚26617 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82467613854720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20257870 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18035642 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1004 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26126781 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192812032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1201427299496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚567 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770640864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1023410176 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20257870 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26148299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚569 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1004 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518875277 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚21520 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11891237376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚21518 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚26348 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 771877888 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80499675455928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20257872 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82467613854720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 218252800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0]X7»2ò- _wandbruntime‚1262  _runtime‚1262vðƒ9óßM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82536318553088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81328748791296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20274746 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26148492 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192798720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1206779867648 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚94 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1207551758848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1199 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20274746 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11593665 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90511868339200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18050750 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1262823119872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81328748791296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46444782 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 _runtime‚1262.919940088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26170036 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79364331631552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91774884257792 % _timestamp‚1.7884891632791295e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81328766794240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46423236 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 216738816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90511844399104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46423236 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79364314592192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80567530711998 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80566743031566 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1206779867648 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6456541 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1263015918592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1202428439374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20274744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11593663 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20274744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26148492 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46444782 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26170036 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82535528658944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18050204sÍ°2é&ôãM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚21546 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11891237376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 771891200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1018907648 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91774884257792 % _timestamp‚1.7884891632791295e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12152995840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1023410176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81328766794240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46423236 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 216738816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90511844399104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46423236 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518875277 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚26348 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79364314592192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1023410176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11891237376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80567530711998 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80566743031566 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1206779867648 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6456541 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1263015918592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1202428439374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20274744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11593663 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20274744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26148492 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚26617 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46444782 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚21544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26170036 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82535528658944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18050204 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82536318553088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 218252800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚488 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81328748791296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 789894144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770641072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚21544 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20274746 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26148492 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192798720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787680432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1206779867648 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚94 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1207551758848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 789894144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1199 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20274746 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11593665 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90511868339200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18050750 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1004 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1262823119872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1004 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81328748791296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46444782 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1004 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1004 _runtime‚1262.919940088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26170036 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79364331631552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚488^ê™]¡2õ- _wandbruntime‚1262  _runtime‚12629w‹[öÕ /loss/FRt1/assignment_loss‚4.327914728359743 0loss/FBt1/assignment_loss‚0.3233034766412207 1loss/SRqqt3/detection_loss‚1.8273029302136372 0loss/FRt3/assignment_loss‚3.6354499838568946 2loss/SRqqt4/assignment_loss‚1.1211616684217005 /loss/FRt2/detection_loss‚2.3807917192578314 epoch‚0 4loss/FBt2/assignment_loss‚2.4496946252838825e-05 1loss/FBt3/assignment_loss‚0.21229167181818412 -loss/FBt1/detection_loss‚2.93240396465574 .loss/FBt2/detection_loss‚5.020558550953865 /loss/FRt2/assignment_loss‚4.694277513027191 0loss/FRt4/assignment_loss‚2.2436716759746727 2loss/SRqqt2/assignment_loss‚2.5087642838125643 0loss/FBt4/assignment_loss‚0.2509252090584616 /loss/FBt4/detection_loss‚0.8756305826169758 $ _timestamp‚1.788489163281517e+09 2loss/SRqqt3/assignment_loss‚2.3760310972141006 1loss/SRqqt1/detection_loss‚2.1907527148723602 /loss/FRt1/detection_loss‚2.5583241541277277 .loss/FRt3/detection_loss‚2.638722621581771 1loss/SRqqt4/detection_loss‚1.0323998319450767 /loss/FBt3/detection_loss‚2.9945454835891723 trainer/global_step‚1199 _runtime‚1262.921698742 /loss/FRt4/detection_loss‚1.8544866028665141 %loss/total_loss‚52.00180048521389 _step‚95 2loss/SRqqt1/assignment_loss‚1.8153865197229953 0loss/SRqqt2/detection_loss‚2.186679003679234[Ðv_÷Ù 2loss/SRqqt1/assignment_loss‚1.8153865197229953 0loss/SRqqt2/detection_loss‚2.186679003679234 /loss/FRt1/assignment_loss‚4.327914728359743 0loss/FBt1/assignment_loss‚0.3233034766412207 1loss/SRqqt3/detection_loss‚1.8273029302136372 0loss/FRt3/assignment_loss‚3.6354499838568946 2loss/SRqqt4/assignment_loss‚1.1211616684217005 /loss/FRt2/detection_loss‚2.3807917192578314 epoch‚0 4loss/FBt2/assignment_loss‚2.4496946252838825e-05 1loss/FBt3/assignment_loss‚0.21229167181818412 -loss/FBt1/detection_loss‚2.93240396465574 .loss/FBt2/detection_loss‚5.020558550953865 /loss/FRt2/assignment_loss‚4.694277513027191 0loss/FRt4/assignment_loss‚2.2436716759746727 2loss/SRqqt2/assignment_loss‚2.5087642838125643 0loss/FBt4/assignment_loss‚0.2509252090584616 /loss/FBt4/detection_loss‚0.8756305826169758 $ _timestamp‚1.788489163281517e+09 2loss/SRqqt3/assignment_loss‚2.3760310972141006 1loss/SRqqt1/detection_loss‚2.1907527148723602 /loss/FRt1/detection_loss‚2.5583241541277277 .loss/FRt3/detection_loss‚2.638722621581771 1loss/SRqqt4/detection_loss‚1.0323998319450767 /loss/FBt3/detection_loss‚2.9945454835891723 trainer/global_step‚1199 _runtime‚1262.921698742 /loss/FRt4/detection_loss‚1.8544866028665141 %loss/total_loss‚52.00180048521389 _step‚95_RÞ5“ø‚(:ˆ ×ÛèÔ‡œàµ gpu.0.gpu‚49.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.193' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë”Tuù‚(:ê ×ÛèÔÁ¸» disk.sda3.out‚ 38.7265625 disk.sda1.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚29189.12353515625! disk.nvme0n1.in‚ 4052.70703125* proc.memory.percent‚17.809748649597168 network.sent‚ 1.581015e+06 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 4052.70703125 disk.dm-0.in‚ 4052.70703125# memory_percent‚85.5762243270874# proc.memory.rssMB‚ 2917.94921875 disk.dm-1.out‚ 38.7265625 disk.drbd1260.out‚ 0.1796875 disk.sda2.out‚0) proc.memory.availableMB‚ 2363.19140625 cpu‚24.62012773836968 proc.cpu.threads‚17) disk./.usagePercent‚78.51241568955076 disk.sda.in‚0 disk.sda.out‚ 38.7265625 disk.sda1.in‚0& disk.nvme0n1.out‚29189.12353515625# disk.dm-0.out‚29189.12353515625 network.recv‚450125$ disk./.usageGB‚1169.891731262207̲µÿ’ú‚(:‡ æÛèÔ›òº­ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚173.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ¬5çtû‚(:é æÛèÔÚÄ…° network.recv‚454695% disk./.usageGB‚1169.7625732421875 disk.drbd1260.out‚ 0.1796875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚29413.57666015625) proc.memory.percent‚17.81158447265625) disk./.usagePercent‚78.50374778645646# disk.dm-0.out‚29413.57666015625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 4052.70703125$ memory_percent‚85.58042049407959 cpu‚24.670692484842128 disk.dm-0.in‚ 4052.70703125 disk.sda3.out‚ 39.08984375 disk.dm-1.out‚ 39.08984375 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 4052.70703125) proc.memory.availableMB‚ 2362.50390625 proc.memory.rssMB‚2918.25 proc.cpu.threads‚17 network.sent‚ 1.616846e+06 disk.drbd1260.in‚0 disk.sda.out‚ 39.08984375( disk.nvme0c0n1.out‚29413.57666015625F>ú>’ü‚(:‡ õÛèÔ¬º…® gpu.0.gpu‚53.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚171.575' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.315 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ³©)xý‚(:í õÛèÔŠå±& disk.nvme0n1.out‚29660.56689453125 disk.dm-0.in‚ 4052.70703125 disk.sda3.in‚0 disk.sda3.out‚ 39.57421875 disk.sda1.out‚0 network.sent‚ 1.625005e+06# disk.dm-0.out‚29660.56689453125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.18359375 disk.sda1.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 2361.66796875 proc.memory.rssMB‚ 2919.03125 network.recv‚459600! disk.nvme0n1.in‚ 4052.70703125 disk.sda.out‚ 39.57421875) proc.memory.percent‚17.81635284423828 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 4052.70703125( disk.nvme0c0n1.out‚29660.56689453125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚85.58552265167236 cpu‚24.722570577965634 disk.dm-1.out‚ 39.57421875) disk./.usagePercent‚78.51371262371923% disk./.usageGB‚1169.9110565185547C=ðk’þ‚(:‡ „ÜèÔÈÒ”­ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚34.0 gpu.0.powerWatts‚113.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7546 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠQwÿ‚(:ì „ÜèÔŽ‘«± disk.nvme0n1.in‚ 4181.5859375# proc.memory.rssMB‚ 2919.14453125 cpu‚24.782507762426853 disk.sda1.out‚0) disk./.usagePercent‚78.53808428497136 disk.sda3.in‚0 disk.drbd1260.out‚ 0.18359375 disk.sda.out‚ 40.05859375 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 4181.5859375 network.recv‚463996$ disk./.usageGB‚1170.274211883545 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0# disk.dm-0.out‚30033.94580078125$ memory_percent‚85.58468818664551( proc.memory.availableMB‚ 2361.8046875 network.sent‚ 1.630999e+06 disk.sda3.out‚ 40.05859375 disk.dm-1.in‚0& disk.nvme0n1.out‚30033.94580078125* proc.memory.percent‚17.817044258117676 proc.cpu.threads‚17 disk.dm-1.out‚ 40.05859375" disk.nvme0c0n1.in‚ 4181.5859375( disk.nvme0c0n1.out‚30033.94580078125ˆ½C=t€bo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPßéÿxbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPž4!]‚bX 5DeviceStatsMonitor.on_validation_batch_start/num_ooms"trainer/global_step2:JP`Þ¹~fƒba >DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPÌ^wm„bh EDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak"trainer/global_step2:JP3Ü[Çx…bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP†¬62o†bj GDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current"trainer/global_step2:JPœŽÁ l‡bg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JP˜Så‚mˆbh EDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current"trainer/global_step2:JP"Š‰–q‰bl IDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current"trainer/global_step2:JP—BtvŠbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP˜C¥¨o‹bj GDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak"trainer/global_step2:JPžI‹rŒbm JDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP²~úqbl IDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated"trainer/global_step2:JP:'ruŽbp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPôF¸¥tbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP/©ºmbh EDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed"trainer/global_step2:JPJÅàn‘bi FDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current"trainer/global_step2:JPyó·~r’bm JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP(ø')q“bl IDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated"trainer/global_step2:JPîö¨o”bj GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated"trainer/global_step2:JP^Þ€?t•bo LDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPóè­âo–bj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JPL“l—bg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JP G÷y˜bt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP‰Ót™bo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPzkayšbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPÿãbþo›bj GDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak"trainer/global_step2:JPn®rjœbe BDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak"trainer/global_step2:JP.Üpbk HDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated"trainer/global_step2:JPö›Cržbm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPl¶kŸbf CDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak"trainer/global_step2:JP“.Œœr bm JDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current"trainer/global_step2:JP  7‰s¡bn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPꤟ|t¢bo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP»#¼Žq£bl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPdþeƒt¤bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP¤ùz m¥bh EDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated"trainer/global_step2:JPž_`„s¦bn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JP¯IǼt§bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPð'›¾x¨bs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP#o¾w|©bw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPÕÛ7%zªbu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPÙZÒl«bg DDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed"trainer/global_step2:JP²Xv¬bq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP2?7Le­b` =DeviceStatsMonitor.on_validation_batch_start/num_device_alloc"trainer/global_step2:JP‚) l®bg DDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP9Tv¯bq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPWë'£k°bf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JPBý‚p±bk HDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed"trainer/global_step2:JP<¨.Py²bt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPoo?|e³b` =DeviceStatsMonitor.on_validation_batch_start/segment.all.peak"trainer/global_step2:JPÛ ¹i´bd ADeviceStatsMonitor.on_validation_batch_start/active.all.allocated"trainer/global_step2:JP6i«nµbi FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP¤Þyv¶bq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP5;Iêf·ba >DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP´#ÅEj¸be BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JP_ä}ïe¹b` =DeviceStatsMonitor.on_validation_batch_start/active.all.freed"trainer/global_step2:JP½åèkºbf CDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak"trainer/global_step2:JP6¥¤Ãy»bt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPë²òmt¼bo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP¡ŸAw½br ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPm„*œp¾bk HDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated"trainer/global_step2:JPlêK®l¿bg DDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed"trainer/global_step2:JP\ÙŠgqÀbl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPÃ.tÁbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPm#žuÂbp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP^8aêqÃbl IDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPô*ÉChÄbc @DeviceStatsMonitor.on_validation_batch_start/segment.all.current"trainer/global_step2:JP¥îvtvÅbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPÀºQuÆbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP"~b¦~Çby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPx)kÈbf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JPï'ˆpÉbk HDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current"trainer/global_step2:JP\·ÇdÊb_ æÖlÞbg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JPY¦ænßbi FDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current"trainer/global_step2:JPnÏfsàbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPªUsábn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPçÏu,pâbk HDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated"trainer/global_step2:JP€ympãbk HDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current"trainer/global_step2:JPç¬PBgäbb ?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections"trainer/global_step2:JP× M7nåbi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JP!ö­kæbf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JPXâ{ñmçbh EDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPûe±Xwèbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP ’Á ébr ODeviceStatsMonitor.on_val4c@ZWidation_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPéü×Ãrêbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPÀ€Ç pëbk HDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed"trainer/global_step2:JPøåƒ]vìbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPÈoíbj GDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed"trainer/global_step2:JPŸ ‚Koîbj GDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current"trainer/global_step2:JPÄz‰ämïbh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JPq{Étðbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP8OºP~ñby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP[aŒæwòbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPìM‚ótóbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP¤û˜@iôbd ADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed"trainer/global_step2:JPåÇF|õbw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPC} Õxöbs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPU=.o÷bj GDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current"trainer/global_step2:JPŠmà_røbm JDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current"trainer/global_step2:JP Õÿýyùbt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP¡á}iúbd ADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams"trainer/global_step2:JPÀ¢…2û- _wandbruntime‚1329  _runtime‚1329ÎÕA*ü»T PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚17429 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚94787262948864 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚26771794 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 597644800 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 11891237376 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 579641856 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 1238802466304 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚26754365 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12152995840 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚6753904 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚93491932542976 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578069228 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚47816603 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 127098368 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚12071515 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚6754089 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518875277 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚488 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚488 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 597644800 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚26348 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 371748352 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚488 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚82031408678736 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚85405380318720 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚18825606 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚85405380318720 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚26771794 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚187 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10558147584 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 1234920241012 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 11891237376 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚83266345959108 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3125 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚17429 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017627277 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1018907648 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚47834034 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚12071517 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚93491908602880 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _step‚96 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1239382108160 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1004 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12152995840 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1023410176 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚17431 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚21062238 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚82031425718096 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1004 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚47834034 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚85404782673920 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚185 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1004 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚47816603 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚26348 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 1239382108160 _runtime‚1329.306247964 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚1208 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚21062240 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚21062238 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 579641856 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1234342171784 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 12152995840 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1004 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚84165998210560 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚84165980207616 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 1295481444352 $ _timestamp‚1.788489229664979e+09 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚85404782673920 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595108588 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518875277 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1023410176 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚488 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚488 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 597644800 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚26348 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 371748352 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚488 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚82031408678736 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚85405380318720 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚18825606 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚85405380318720 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚26771794 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚187 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10558147584 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 1234920241012 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 11891237376 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚83266345959108 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3125 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚17429 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017627277 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1018907648 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚47834034 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1018907648 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚12071517 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚93491908602880 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _step‚96 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1239382108160 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1004 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12152995840 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1023410176 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚17431 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚21062238 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚82031425718096 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1004 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚47834034 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚85404782673920 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚185 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1004 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚47816603 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚26348 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 1239382108160 _runtime‚1329.306247964 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚1208`-jñoþbj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPsÜ-Üuÿbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPF~k€bf CDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed"trainer/global_step2:JPD×Éòsbn KDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPcÞù:p‚bk HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP£ú¿»kƒbf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JP×Änrm„bh EDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated"trainer/global_step2:JP%Ët…bo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP_!›Mk†bf CDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPHš[tw‡br ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPÐ-Ñ5lˆbg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JP6ðrw‰br ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP)óegŠbb ?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed"trainer/global_step2:JPq(07m‹bh EDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current"trainer/global_step2:JP{mOÆkŒbf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPMv~kbf CDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed"trainer/global_step2:JP³ø³jŽbe BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JPv¾åTjbe BDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed"trainer/global_step2:JPEpm{zbu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP½=xŠr‘bm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JP3Ø\’d’b_ t¨bo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPÏÔåYw©br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPîaBpªbk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JPj¹Èóp«bk HDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPòHq¬bl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JPßîÜtk­bf CDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated"trainer/global_step2:JPF7þ€m®bh EDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak"trainer/global_step2:JPìÔws¯bn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPÄ0É£r°bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPD",i±bd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JP—îr²bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPQ{qo³bj GDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPH#óp´bk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP÷œscµb^ ;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc"trainer/global_step2:JPI}Š[¶bV 3DeviceStatsMonitor.on_validation_batch_end/num_ooms"trainer/global_step2:JPÜÃcÏl·bg DDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current"trainer/global_step2:JPóž q¸bl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPPá o¹bj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPT4 ¤qºbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP:áTýw»br ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPµ%Öp¼bk HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JP®Üèl½bg DDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPŽ@ɼr¾bm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPŸAˆ¨q¿bl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP‰>jÀbe BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JP$ª?vÁbq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP$Ï£¶oÂbj GDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current"trainer/global_step2:JPâ‡sÃbn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP›Ž¬hÄbc @DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak"trainer/global_step2:JPî * fÅba >DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JPÚŸ—wÆbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP·!`bÇb] :DeviceStatsMonitor.on_validation_batch_end/num_device_free"trainer/global_step2:JPå¡ÙêuÈbp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPæÏ« ;Ébg DDeviceStatsMonitor.on_validation_batch_end/requested;œ!çnÓbi FDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current"trainer/global_step2:JP\4`úuÔbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPRâóiÕbd ADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak"trainer/global_step2:JPâþ®‚fÖba >DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPèhm×bh EDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current"trainer/global_step2:JPüÑômØbh EDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current"trainer/global_step2:JPysLünÙbi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JPC»RêmÚbh EDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed"trainer/global_step2:JPú–½oÛbj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JP2égÜbb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JPÊÖŒýrÝbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPÄïÀPpÞbk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JP¸š¥bkßbf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JPÑ_f¡oàbj GDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated"trainer/global_step2:JP꫼Œ|ábw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP5Àkxâbs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP"ã›zgãbb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JP0:­ooäbj GDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated"trainer/global_step2:JPÆŽ?>jåbe BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JP›ÁÀiæbd ADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak"trainer/global_step2:JPH²I·nçbi FDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed"trainer/global_step2:JPwjrèbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP5weéb` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JPíqOpêbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPm"ñrëbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPS ‰‡uìbp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPÐ…£tíbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPQ[$yhîbc @DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated"trainer/global_step2:JPŸ&vøjïbe BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JPöEö³rðbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPE«=7sñbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPÐ7Û>nòbi FDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed"trainer/global_step2:JPô”¹móbh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JPµàC¨qôbl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPÀÉ)brõbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPy{¡¾eöb` =DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections"trainer/global_step2:JP¬bn÷bi FDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated"trainer/global_step2:JPìq8èbøb] :DeviceStatsMonitor.on_validation_batch_end/active.all.peak"trainer/global_step2:JPôñTé2ù- _wandbruntime‚1330  _runtime‚1330 boL)úÆR [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 579641856 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1004 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚516 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚488 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚26780088 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚488 trainer/global_step‚1208 % _timestamp‚1.7884892309310741e+09 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚47855300 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 597644800 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚18838791 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚93538910688256 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 1295819094528 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚488 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚26348 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚26762659 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚17431 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1239267214848 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚12079495 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1023410176 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1004 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚26762659 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 579641856 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 1239267214848 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚21075210 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚26348 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚26617 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1018907648 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚18838602 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 151038464 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚83309104960168 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚85451313066496 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚84211466209792 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11891237376 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1235383401976 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚47837869 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚189 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚516 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚21075210 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 127098368 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚26617 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 _step‚97 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 578069228 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚83308509851580 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚21075212 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚26780088 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚6759294 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚85450715421696 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚94834856881152 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚93538886748160 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595108588 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 11518875277 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12152995840 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 597644800ö#ÍP)ûÊR SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚21075210 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚26348 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1023410176 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1018907648 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚26617 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1018907648 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚18838602 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 151038464 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚83309104960168 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚85451313066496 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚84211466209792 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11891237376 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1235383401976 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚47837869 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚189 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚516 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚21075210 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 127098368 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚26617 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 _step‚97 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 578069228 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚83308509851580 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚21075212 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚26780088 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚6759294 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚85450715421696 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚94834856881152 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚93538886748160 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595108588 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 11518875277 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12152995840 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 597644800 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 579641856 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1004 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚516 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚488 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚26780088 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚488 trainer/global_step‚1208 % _timestamp‚1.7884892309310741e+09 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚47855300 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 597644800 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚18838791 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚93538910688256 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 1295819094528 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚488 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚26348 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚26762659 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚17431 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1239267214848 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚12079495 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1023410176 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1004 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚26762659 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 579641856 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 1239267214848a†@ížü‚(:“ “ÜèÔûã® gpu.0.gpu‚43.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚171.497' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.299400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óz¤ sý‚(:è “ÜèÔ˜°¸¯" memory_percent‚85.585618019104) proc.memory.availableMB‚ 2361.65234375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚78.54712339899032 disk.dm-1.out‚ 40.58984375 disk.sda2.in‚0 cpu‚24.837343739420707 network.recv‚467519 disk.sda3.out‚ 40.58984375 disk.sda.out‚ 40.58984375# disk.nvme0c0n1.in‚ 4209.52734375# proc.memory.rssMB‚ 2919.16796875* proc.memory.percent‚17.817187309265137 disk.sda.in‚0 network.sent‚ 1.634227e+06% disk./.usageGB‚1170.4089012145996 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.1875' disk.nvme0c0n1.out‚30461.9052734375 disk.sda2.out‚0! disk.nvme0n1.in‚ 4209.52734375% disk.nvme0n1.out‚30461.9052734375 disk.dm-0.in‚ 4209.52734375" disk.dm-0.out‚30461.9052734375e+S3NþbI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP*:NÿbI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPJÆbE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP»Ùv–HÇbC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPf'¦HÈbC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPŒ…bHÉbC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPöëèNÊbI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPŒÚ®WNËbI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPQš„HJÌbE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP¢_91HÍbC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP/;ÒHÎbC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JP}ÒÙ§JÏbE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÖŸËHÐbC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP±7DfJÑbE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP¡ëHÒbC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPº2ŽNÓbI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPΔ6NÔbI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP1†N@Õb; particle/accuracy_2_of_3"trainer/global_step2:JPÍŠëYNÖbI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP58fÎJ×bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPSwJØbE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPB-Ñ‹JÙbE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP¢¸æJÚbE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPAÚr#NÛbI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP~YÎNÜbI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP··‹ÑJÝbE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPM8²JÞbE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÞÒ2)NßbI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP»Êy«NàbI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPû×¢JábE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPâãNâbI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPJwñJãbE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP^Jï„NäbI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP¹ÞˆHHåbC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPѹ*¢JæbE "Purity/3FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPQwHçbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPí½Ë„JèbE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPÏxqcHébC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¨¦nNêbI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP­C¯,JëbE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP»¥õNìbI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPꚺLHíbC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP8.û²@îb; particle/accuracy_3_of_6"trainer/global_step2:JP…^‘NïbI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP¬"ÊJðbE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPìsêJHñbC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPœ‡ØDJòbE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPh²ˆNóbI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPØpñÊJôbE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPá¨RHõbC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP9í$=JöbE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP ~Û?H÷bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP·œp¶HøbC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPàÑèÞ@ùb; particle/accuracy_1_of_2"trainer/global_step2:JP-øBHúbC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP[`ãÑJûbE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPT¹&êJübE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPˆÕQ JýbE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP¤"œMNþbI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP¦m@NÿbI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPøŠã@€ b; particle/accuracy_1_of_5"trainer/global_step2:JP棞w; b6 jet/accuracy_3_of_5"trainer/global_step2:JP•ÒMH‚ bC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPbsvHƒ bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPybñeJ„ bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPt\{GN… bI &Purity/0FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPM&I{H† bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP=‘ŠN‡ bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPzÉKNˆ bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP¬Ê H‰ bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP…â“aJŠ bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP³BiH‹ bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPI"çNŒ bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP˜öTˆJ bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPDj¦NŽ bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPjç‘ËJ bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¿œ0•N bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP_™‰;N‘ bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPiÞN’ bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP¡èqH“ bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPBã8ûN” bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP¦€@• b; particle/accuracy_1_of_3"trainer/global_step2:JPåÄ ãH– bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP¨lu«J— bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPl«ùgJ˜ bE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPg›gr;™ b6 jet/accuracy_3_of_7"trainer/global_step2:JPW™$™Nš bI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPÂtKJ› bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP&(ŠˆNœ bI &Purity/0FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPàÓÍóJ bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP˜ä+ÐHž bC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP†› NŸ bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP.â}H  bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPp B*J¡ bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP#H¿ bC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPâW7NÀ bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP¡fk¡NÁ bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP}êŽJ bE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP“úUOJà bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP[àÊÃHÄ bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPäýnHÅ bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPôâ~ïNÆ bI &Purity/*FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP¬ËµHÇ bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP ¬óùNÈ bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPдXHÉ bC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPΣNÊ bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPèC‚‘NË bI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPD$* JÌ bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPYÊ´NÍ bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPsü#þNÎ bI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPK¬–IJÏ bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPÔ®AJÐ bE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP(S»ÑJÑ bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPõ8NÒ bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPérFiHÓ bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPLæ¡ËJÔ bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPJÌC¯@Õ b; particle/accuracy_6_of_6"trainer/global_step2:JP~sçªJÖ bE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPT‘]NN× bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP6N%#HØ bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP `éJÙ bE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÔ’sÃHÚ bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP»þÜÎHÛ bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPB–Q8JÜ bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP‹-U“HÝ bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPP^`àJÞ bE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPQã}Hß bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPgöCHà bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÕ©PNá bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP2 îäNâ bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPtÐD)Jã bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPË ŸHä bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPF$aJå bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP€«J¬Næ bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPû(_Nç bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP-‘úrNè bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP´”'Né bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP\|Á«Jê bE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÈàþ8Jë bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¬åKJì bE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP¬Æ¸`Ní bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP#n“Nî bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP8ÓµNï bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPñŽÐNð bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPS±Ù+Hñ bC Purity/2FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPmQ ¬Hò bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPuFêŸHó bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPÄž%Jô bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPËJï¢Jõ bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP¼mJö bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPŒÒ*IN÷ bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPî5{hJø bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP³¥ÆEHù bC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JP{çÆNú bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP/^¸Nû bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPF pJHü bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPƧˆJý bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPóWÍJþ bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP\ÏWHHÿ bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÉNuZJ€ bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP¾ózxN bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPjeCl@‚ b; particle/accuracy_2_of_5"trainer/global_step2:JP2KæbJƒ bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP'ÄÇ;„ b6 jet/accuracy_2_of_6"trainer/global_step2:JPQ=Vl;… b6 jet/accuracy_6_of_6"trainer/global_step2:JP­KäâH† bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPì䦰N‡ bI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPåvbNJˆ bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPCz(¶J‰ bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP\ËJŠ bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP‘F©ÓN‹ bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÑœ¤NŒ bI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPxÝ7,; b6 jet/accuracy_1_of_4"trainer/global_step2:JP_@?®NŽ bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP„{ÿrH bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP¤Žë·H bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP8ÿ *H‘ bC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPF½†H’ bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP>o˜üJ“ bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP¿N×SN” bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP{ÜñJ• bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPa7øN– bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP}ŠŠYH— bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP«P2;˜ b6 jet/accuracy_2_of_4"trainer/global_step2:JP•B`cH™ bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP;`ŠôJš bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP ùÙ6H› bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPæ¥ÓNœ bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPL†î¤; b6 jet/accuracy_1_of_3"trainer/global_step2:JP)6:;ž b6 jet/accuracy_3_of_4"trainer/global_step2:JP”{¼HŸ bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPèõJ  bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPo}ÁøN¡ bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPŒ5)—J¢ bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¯©N£ bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPAŠN¤ bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP56¥yH¥ bC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPì²"ä;¦ b6 jet/accuracy_4_of_7"trainer/global_step2:JP.¶X»H§ bC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPT0<…N¨ bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP0eNJ© bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPåÛ=÷Hª bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP7jzN« bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPKƒªN¬ bI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP³-AH­ bC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP´s J® bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÕ@QRN¯ bI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPb±û N° bI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPЯ¿DN± bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP{Ö H² bC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPmbWg;³ b6 jet/accuracy_5_of_7"trainer/global_step2:JP‡]aÕ;´ b6 jet/accuracy_7_of_7"trainer/global_step2:JPojð’Jµ bE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPx½H¶ bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP³çƒýN· bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP=«ÆaJ¸ bE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPÀþ©J¹ bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP;2åJº bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP?<"H» bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP‰…MSH¼ bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP4SÓÓH½ bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPóRéN¾ bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPyF˜ÈJ¿ bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÜÇfvJÀ bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP2£ÈV@Á b; particle/accuracy_7_of_7"trainer/global_step2:JP˜cÒ*N bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP¿'_Hà bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPüŸJÄ bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP÷ØÜHÅ bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPÍãÇ×JÆ bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP[mxNÇ bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPÖŒïJÈ bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPííŽÀNÉ bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPteÓJÊ bE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP j³…JË bE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP/¡á3HÌ bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JP“ÌûáJÍ bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPµŽ¯ªHÎ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP–³£JÏ bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPɆ:ÄNÐ bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP5›oÔNÑ bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPµn1@Ò b; particle/accuracy_5_of_7"trainer/global_step2:JP>7PJÓ bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPSÅ}8HÔ bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÿW:NHÕ bC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPvßš÷@Ö b; particle/accuracy_2_of_4"trainer/global_step2:JPñÙ9AH× bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP7 ïNØ bI &Purity/4FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPàwÞNÙ bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP”‡l\HÚ bC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPé$ŸJÛ bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPxóÍJÜ bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP(¬€ºJÝ bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP›>®;Þ b6 jet/accuracy_2_of_3"trainer/global_step2:JPœà1*@ß b; particle/accuracy_1_of_1"trainer/global_step2:JPˆ•uJà bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPN”ŸJá bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP,pÔKHâ bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP'âNã bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP˜) Hä bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPi4'`@å b; particle/accuracy_5_of_6"trainer/global_step2:JP„ƒ³Jæ bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPY.Nç bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPNêTÁHè bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP×¾Jé bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPÅîHê bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPë¾ßNë bI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP6ܽNì bI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPË~Jí bE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÖ€ŠHî bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPÎѯNï bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP²>;;ð b6 jet/accuracy_4_of_4"trainer/global_step2:JPÞI?ª@ñ b; particle/accuracy_3_of_7"trainer/global_step2:JP¤Â$i@ò b; particle/accuracy_4_of_7"trainer/global_step2:JP£•Á Nó bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP!OüpNô bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPûw|´Jõ bE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÀšFHö bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPUåKNJ÷ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPï<ë,Nø bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPàñ¾¨Jù bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP þ“4;ú b6 jet/accuracy_5_of_6"trainer/global_step2:JP©Å‘Hû bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPe8;ü b6 jet/accuracy_2_of_5"trainer/global_step2:JP}ØéHý bC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPä$‚…@þ b; particle/accuracy_6_of_7"trainer/global_step2:JPÖâ!òJÿ bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPq}¤5N€ bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP//!ÂN bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÃÑ!N‚ bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP£éðNƒ bI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP¯K6N„ bI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÐÍ:Ø@… b; particle/accuracy_3_of_4"trainer/global_step2:JP|Pã»J† bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPdÚJ‡ bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP€`¯tJˆ bE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPY/öžJ‰ bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP/Æ<:HŠ bC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPÇhx H‹ bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP”«ùJŒ bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP;|MKH bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP&5/NŽ bI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP)—|N bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP·At; b6 jet/accuracy_1_of_1"trainer/global_step2:JP¯"äH‘ bC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPíÌH’ bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP x/J“ bE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPÍ,PJ” bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP‚ȵJ• bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¦ø*úN– bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPä²C³N— bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPîaugN˜ bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPw{;Ñ;™ b6 jet/accuracy_3_of_3"trainer/global_step2:JPÜÏ`Jš bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPhÇ<J› bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP"Ç’šNœ bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPšÑþJ bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPƒÙ8¾J« bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP_ ÑfH¬ bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPEuÛH­ bC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP-þž©J® bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPGÁJ¯ bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPás­N° bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPMj‹ßJ± bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¯}YN² bI &Purity/4FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP€:¬×;³ b6 jet/accuracy_1_of_5"trainer/global_step2:JPôvµ²H´ bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPšÙȯHµ bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÿz9%H¶ bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPªÑ‹cJ· bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÃËïN¸ bI &Purity/3FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPŽA¾H¹ bC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPƒ¦b‘Hº bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPPRóoH» bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÁSqLN¼ bI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP/¿Á_N½ bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPi˜*SJ¾ bE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPiÉü¨N¿ bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP;øÝ@JÀ bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP[µtNÁ bI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP²ÈH bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPú—”Jà bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP&ë®ÊNÄ bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPŸƒÁ NÅ bI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPj¨¿^JÆ bE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP|N¤@Ç b; particle/accuracy_1_of_7"trainer/global_step2:JPz(H‹HÈ bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP$+¥jJÉ bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP'©wUJÊ bE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÁ.ë=NË bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP¢R}£;Ì b6 jet/accuracy_1_of_2"trainer/global_step2:JPìë¾¹NÍ bI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP׉JÎ bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPúUqxHÏ bC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP]˨JÐ bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÀо§HÑ bC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPwAaJÒ bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPD¨ITJÓ bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPŒ¾0HÔ bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPA˜áHÕ bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPR6·hHÖ bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPqÊ(J× bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP=Y §JØ bE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPrBJÙ bE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPï‚ݽJÚ bE "Purity/1FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPë6?(NÛ bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP"vÈJÜ bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP^jª$JÝ bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÐ'cµHÞ bC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPÅ[äJß bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¶O®EJà bE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPïJá bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP—/ÂqJâ bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPJ+#¬Nã bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPŠßENä bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPZJå bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPpXåíJæ bE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP5€œCJç bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP#WÔŠJè bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP§ø Né bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP»7ÊNê bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP,¼éÛNë bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP ¥Œ|Hì bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPr?KHí bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPçC¶Hî bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPCŒ;ï b6 jet/accuracy_5_of_5"trainer/global_step2:JP­µäÜ<ð b7 particle/sensitivity"trainer/global_step2:JP1v¸Jñ bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP€°AõNò bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÒíwNó bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP kòHô bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPç¦1„Hõ bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPÔp½;Nö bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPƒKþN÷ bI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÔagHø bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP5Î!*Jù bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPp1Nú bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP¤É~Nû bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP»ú#CHü bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP/ó¼Jý bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP€ØHþ bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPY{ ÊNÿ bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPxŸ7šJ€ bE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÑÖ cN bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPE=ð:H‚ bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPßÿµ=Nƒ bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPå`H„ bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPoª>ýN… bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPå‡vHN† bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPgž`-J‡ bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPs–8Hˆ bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPÆÜâöJ‰ bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPæ:\pHŠ bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPdýQJN‹ bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP©#lGŒ bB validation_average_jet_accuracy"trainer/global_step2:JPzÃO; b6 jet/accuracy_1_of_7"trainer/global_step2:JPë4úËJŽ bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÇáßN bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÅa‹ËJ bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPÛ¨¿4H‘ bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPWÌ´J’ bE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP}ÉÐ,H“ bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP´@|ñN” bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP q`N• bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP÷åí@– b; particle/accuracy_4_of_6"trainer/global_step2:JPSc,ØH— bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPA,×!J˜ bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPNøšJ™ bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP.^¼nJš bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPåü˜BJ› bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP–Kö²Jœ bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÞßN bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPÉ`‘Hž bC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPü¹y¸HŸ bC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP`RN  bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPO+KJ¡ bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPNÔ,ßJ¢ bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP2lVüJ£ bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPŠÕDJN¤ bI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPPW H¥ bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPE™úJ¦ bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP ‹ñH§ bC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP¯ˆêJ¨ bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPŸÊôH© bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPÊwÿþNª bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPä~+&H« bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPH×ÄåN¬ bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPèþH­ bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPì­ÓJ® bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPæp‚H¯ bC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP›ŸµN° bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPàÅLJ± bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPMQÍœN² bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPË<ÁH³ bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP I$N´ bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP5/¥@µ b; particle/accuracy_1_of_6"trainer/global_step2:JPÇÚnH¶ bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP&ƒ ö;· b6 jet/accuracy_2_of_7"trainer/global_step2:JP‚'†N¸ bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP2}p*N¹ bI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP¸t·/Jº bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPКº1J» bE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPƒ_`H¼ bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPeâãH½ bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP¬VtœJ¾ bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPF–RN¿ bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÔílÑNÀ bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP0wÉJÁ bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP¨eÑH bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP Ã-AJà bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP-£Ï©JÄ bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPv¹ö·HÅ bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPm>.®NÆ bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPE0FŽHÇ bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP¦TšQJÈ bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPcÄØHÉ bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPõ,“HHÊ bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPgdcHË bC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPàJU«NÌ bI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP9y0Ù8Í b3 particle/f_score"trainer/global_step2:JP:—ð`JÎ bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÒŒJÏ bE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP^&_JÐ bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPãaŽJÑ bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP ÝæcHÒ bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPp`ÙNÓ bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPm7æJÔ bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPWŒNJÕ bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP*û†HÖ bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP…æSWH× bC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP®—/±NØ bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPäʯ HÙ bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPð4ûNÚ bI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPGáÎJÛ bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÎ;‹NÜ bI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP\¦çPJÝ bE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPPm/NÞ bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPиž@ß b; particle/accuracy_1_of_4"trainer/global_step2:JP8Š\™@à b; particle/accuracy_2_of_6"trainer/global_step2:JP¼½µHá bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPÈù•ÉHâ bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP/ŸÂÊHã bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP@X« Jä bE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPrº"QNå bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP.KHæ bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPâJÔtJç bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP¯iN<è b7 particle/specificity"trainer/global_step2:JPÉXÃHé bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPÔÀjëJê bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPUuÖJë bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPuI¥éHì bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP²j¢³Ní bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP1§oÌJî bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP¯èL^Nï bI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP'öÂÖJð bE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPo÷:`Jñ bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP†G5”Nò bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP„KÞ Nó bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPœ˜úHô bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP~%Nõ bI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP·ñNö bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPºNà@H÷ bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPXMòJø bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPrP3Jù bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP î@ú b; particle/accuracy_4_of_4"trainer/global_step2:JP-†Jû bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPöõÉ£Jü bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPgÎöÊHý bC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPÖÁiòHþ bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPŠKTíNÿ bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPøãqèJ€ bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPü6-H bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP4’IJ‚ bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPƒ•zÏHƒ bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPc‰$J„ bE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP’kÏ(H… bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPùÈ…˜H† bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP:@×ÂJ‡ bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP—,j\Hˆ bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPV éJ‰ bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPŽš%°JŠ bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPK¥#H‹ bC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPC´ÐÊHŒ bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPŠ–EÑN bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP>ûÌNŽ bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP”õþ®N bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPcÛÏïJ bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÖÇ´J‘ bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP7*ÿ_H’ bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP†SøfN“ bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPmkc H” bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPÆ3ÎON• bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP¸ò H– bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÑP³mN— bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP»…0cJ˜ bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPß·€;™ b6 jet/accuracy_4_of_6"trainer/global_step2:JP0øù(Hš bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP¤mµ’N› bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÇýÞNœ bI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPmà1DJ bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP“±°ÖNž bI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPØæ ¼@Ÿ b; particle/accuracy_3_of_5"trainer/global_step2:JPôH»H  bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP„ 5H¡ bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JP4®1@¢ b; particle/accuracy_2_of_2"trainer/global_step2:JP+0 J£ bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP&¤ílJ¤ bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPôÊYbH¥ bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPÝufJ¦ bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP£v;H§ bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP‰á´N¨ bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP„ØDÅN© bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPSý‚Jª bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP³ù+ªH« bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP3ÄóN¬ bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPâà]N­ bI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÜvïóN® bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP7«ú H¯ bC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP4¡¶ÐJ° bE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPŠ¥LN± bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPýu΋J² bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP„êacN³ bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP 9dH´ bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPG?pNµ bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP/· ëN¶ bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP¹;· b6 jet/accuracy_1_of_6"trainer/global_step2:JPr¹¿H¸ bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPú‡øÇN¹ bI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPQÈ8Hº bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPùGFáJ» bE "Purity/1FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP_ŸìN¼ bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP!ãB?N½ bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP1ôi½N¾ bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPŸ9ÂøH¿ bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP„}mâHÀ bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPûj·uJÁ bE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP21[3J bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPðoëcNà bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPb2ÞåNÄ bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP¨á¤qHÅ bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPtÈNÆ bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP@›@„NÇ bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP9½P;È b6 jet/accuracy_4_of_5"trainer/global_step2:JPdÉ<–HÉ bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPëÎhT@Ê b; particle/accuracy_2_of_7"trainer/global_step2:JPí(?JË bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPÕ¯Þ×NÌ bI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÛ´mÍ2Í - _wandbruntime‚1346  _runtime‚1346T5¥€«/Î Õ¥ 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3408038318157196 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9889018535614014 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9738159775733948 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.028777431696653366 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚0.5 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20783251523971558 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2885521650314331 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.012983269058167934 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9887150526046753 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.013631781563162804 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_7‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4464989900588989 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13234125077724457 particle/accuracy_2_of_4‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4020598828792572 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2747417390346527 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4019899070262909 *jet/accuracy_2_of_3‚0.2068169116973877 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1711014062166214 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.45064103603363037 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13261841237545013 particle/accuracy_5_of_6‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3430873155593872 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.25450190901756287 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12912198901176453 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0008818342466838658 particle/accuracy_3_of_7‚1 particle/accuracy_4_of_7‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9859843850135803 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3006761968135834 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.40414518117904663 jet/accuracy_5_of_6‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13060283660888672 +jet/accuracy_2_of_5‚0.40334102511405945 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 particle/accuracy_6_of_7‚1 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.41523176431655884 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.005931040737777948 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3950064182281494 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20228435099124908 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 *jet/accuracy_1_of_1‚0.4266137182712555 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07446808367967606 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0833333358168602 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.32863983511924744 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_3‚0.01576186530292034 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4018232524394989 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.18518516421318054 *"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3942019045352936 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20567402243614197 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.21052630245685577 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.40912023186683655 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3674794137477875 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.24267242848873138 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9130567908287048 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1302666813135147 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.26962581276893616 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20910494029521942 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.34329938888549805 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06354985386133194 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 _runtime‚1346.410131896 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2807539403438568 particle/accuracy_1_of_7‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 _step‚98 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 *jet/accuracy_1_of_2‚0.6760116815567017 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.21875 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2938997447490692 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2138611227273941 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9781931042671204 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3523198366165161 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13565963506698608 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6848633885383606 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.37079691886901855 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4050525724887848 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1878766566514969 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 jet/accuracy_5_of_5‚0 ,particle/sensitivity‚0.28585827350616455 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2687326669692993 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15074774622917175 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.31790125370025635 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9957020282745361 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.18518519401550293 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9877644181251526 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.010825375095009804 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 6validation_average_jet_accuracy‚0.3862591087818146 jet/accuracy_1_of_7‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.36367252469062805 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3153044879436493 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12065709382295609 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.41025716066360474 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.322172611951828 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04960005730390549 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005108923185616732 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_pu;D;1crity‚1 +"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.25 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.34463322162628174 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9804848432540894 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3310234248638153 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5909090638160706 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1419752985239029 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.034535616636276245 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9882856607437134 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 jet/accuracy_2_of_7‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4093870520591736 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14840655028820038 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004251700360327959 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1434684693813324 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4010672867298126 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15418069064617157 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.36376482248306274 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03274039551615715 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11657683551311493 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333730697632 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2027558833360672 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9954574108123779 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4209730625152588 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2618618309497833 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_6‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9870233535766602 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30694445967674255 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18667206168174744 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 *particle/specificity‚0.947272539138794 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2601167857646942 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04136905074119568 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_4‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.27259135246276855 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9804546236991882 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.39868828654289246 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3296566903591156 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.36512985825538635 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11110799014568329 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472805023193 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9885786175727844 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.306919664144516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 jet/accuracy_4_of_6‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 particle/accuracy_2_of_2‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9959346055984497 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03448275849223137 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.36742421984672546 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18409642577171326 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.45543399453163147 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,jet/accuracy_4_of_5‚0.002016128972172737 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35733741521835327 particle/accuracy_5_of_5‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.2560422420501709 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18448565900325775 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884892467673743e+09 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3333333134651184 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018424035981297493 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16143234074115753 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 epoch‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.39868828654289246 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.34704646468162537 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19687116146087646 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3401162922382355 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 ,jet/accuracy_3_of_6‚0.057692307978868484 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9839825630187988 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.3571428656578064 jet/accuracy_6_of_7‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3376530706882477 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11398635059595108 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07170543074607849 particle/accuracy_4_of_5‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.3245614171028137 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3403650224208832 particle/accuracy_3_of_3‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3266666531562805 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 )loss/val_total_loss‚48.60559262035512 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33648374676704407 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9826883673667908 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.33908048272132874 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11110799014568329 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.33819153904914856 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.23764365911483765 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.018906811252236366 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.30891287326812744 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3482804596424103 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.36018502712249756 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19227556884288788 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9914911985397339 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3625016212463379 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12222222238779068 particle/accuracy_3_of_6‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3637543320655823 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18945415318012238 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3607456088066101 particle/accuracy_1_of_2‚1 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3472222089767456 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13820336759090424 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_5‚1 trainer/global_step‚1208 +jet/accuracy_3_of_5‚0.10202892869710922 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07521858811378479 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3397321403026581 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13350501656532288 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.36048218607902527 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2601364552974701 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9608476758003235 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1944444477558136 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_7‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8152943849563599 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.36773836612701416 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2767857015132904 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.450940877199173 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011669651605188847 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.560606062412262 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3692082464694977 'particle/accuracy‚0.843982458114624 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17934976518154144 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34223392605781555 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09355480223894119 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3193840980529785 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.988025426864624 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.0905056968331337 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982894659042358 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.34761902689933777 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3799999952316284 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.32279542088508606 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4486111104488373 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9847952723503113 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6221491098403931 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.15708811581134796 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9964895844459534 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2222222238779068 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17905284464359283 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3621920347213745 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9840806722640991 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9872235655784607 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09254488348960876 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18430252373218536 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3660241663455963 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333730697632 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15454547107219696 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17916420102119446 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12207305431365967 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9846896529197693 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.23076923191547394 jet/accuracy_6_of_6‚0 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1816524714231491 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0014550264459103346 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.3265954554080963 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8668577671051025 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31111112236976624 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.32303160429000854 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.3399425446987152 *jet/accuracy_2_of_4‚0.2949811518192291 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.916075587272644 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23974908888339996 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4621114134788513 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 *jet/accuracy_1_of_3‚0.7975584864616394 +jet/accuracy_3_of_4‚0.04360226169228554 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9944405555725098 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3869580030441284 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_4_of_7‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14149919152259827 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5099726319313049 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.29139429330825806 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 jet/accuracy_5_of_7‚0 jet/accuracy_7_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1Ÿx­uÁÏ Ù¥ + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3942019045352936 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20567402243614197 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.21052630245685577 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.40912023186683655 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3674794137477875 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.24267242848873138 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9130567908287048 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1302666813135147 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.26962581276893616 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20910494029521942 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.34329938888549805 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06354985386133194 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 _runtime‚1346.410131896 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2807539403438568 particle/accuracy_1_of_7‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 _step‚98 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 *jet/accuracy_1_of_2‚0.6760116815567017 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.21875 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2938997447490692 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2138611227273941 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9781931042671204 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3523198366165161 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13565963506698608 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6848633885383606 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.37079691886901855 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4050525724887848 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1878766566514969 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 jet/accuracy_5_of_5‚0 ,particle/sensitivity‚0.28585827350616455 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2687326669692993 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15074774622917175 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.31790125370025635 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9957020282745361 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.18518519401550293 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9877644181251526 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.010825375095009804 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 6validation_average_jet_accuracy‚0.3862591087818146 jet/accuracy_1_of_7‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.36367252469062805 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3153044879436493 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12065709382295609 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.41025716066360474 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.322172611951828 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04960005730390549 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005108923185616732 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.25 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.34463322162628174 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9804848432540894 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purî÷ð—vity/2FRt*SRqqt1FBt/FRt_purity‚0.3310234248638153 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5909090638160706 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1419752985239029 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.034535616636276245 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9882856607437134 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 jet/accuracy_2_of_7‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4093870520591736 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14840655028820038 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004251700360327959 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1434684693813324 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4010672867298126 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15418069064617157 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.36376482248306274 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03274039551615715 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11657683551311493 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333730697632 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2027558833360672 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9954574108123779 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4209730625152588 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2618618309497833 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_6‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9870233535766602 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30694445967674255 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18667206168174744 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 *particle/specificity‚0.947272539138794 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2601167857646942 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04136905074119568 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_4‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.27259135246276855 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9804546236991882 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.39868828654289246 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3296566903591156 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.36512985825538635 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11110799014568329 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472805023193 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9885786175727844 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.306919664144516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 jet/accuracy_4_of_6‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 particle/accuracy_2_of_2‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9959346055984497 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03448275849223137 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.36742421984672546 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18409642577171326 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.45543399453163147 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,jet/accuracy_4_of_5‚0.002016128972172737 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35733741521835327 particle/accuracy_5_of_5‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.2560422420501709 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18448565900325775 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884892467673743e+09 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3333333134651184 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018424035981297493 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16143234074115753 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 epoch‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.39868828654289246 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.34704646468162537 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19687116146087646 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3401162922382355 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 ,jet/accuracy_3_of_6‚0.057692307978868484 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9839825630187988 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.3571428656578064 jet/accuracy_6_of_7‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3376530706882477 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11398635059595108 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07170543074607849 particle/accuracy_4_of_5‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.3245614171028137 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3403650224208832 particle/accuracy_3_of_3‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3266666531562805 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 )loss/val_total_loss‚48.60559262035512 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33648374676704407 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9826883673667908 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.33908048272132874 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11110799014568329 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.33819153904914856 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.23764365911483765 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.018906811252236366 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.30891287326812744 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3482804596424103 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.36018502712249756 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19227556884288788 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9914911985397339 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3625016212463379 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12222222238779068 particle/accuracy_3_of_6‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3637543320655823 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18945415318012238 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3607456088066101 particle/accuracy_1_of_2‚1 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3472222089767456 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13820336759090424 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_5‚1 trainer/global_step‚1208 +jet/accuracy_3_of_5‚0.10202892869710922 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07521858811378479 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3397321403026581 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13350501656532288 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.36048218607902527 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2601364552974701 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9608476758003235 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1944444477558136 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_7‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8152943849563599 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.36773836612701416 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2767857015132904 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.450940877199173 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011669651605188847 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.560606062412262 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3692082464694977 'particle/accuracy‚0.843982458114624 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17934976518154144 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.34223392605781555 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09355480223894119 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3193840980529785 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.988025426864624 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.0905056968331337 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982894659042358 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.34761902689933777 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3799999952316284 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.32279542088508606 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4486111104488373 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9847952723503113 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6221491098403931 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.15708811581134796 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9964895844459534 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2222222238779068 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17905284464359283 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3621920347213745 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9840806722640991 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9872235655784607 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09254488348960876 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18430252373218536 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3660241663455963 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333730697632 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15454547107219696 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17916420102119446 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12207305431365967 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9846896529197693 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.23076923191547394 jet/accuracy_6_of_6‚0 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1816524714231491 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0014550264459103346 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.3265954554080963 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8668577671051025 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31111112236976624 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.32303160429000854 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.3399425446987152 *jet/accuracy_2_of_4‚0.2949811518192291 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.916075587272644 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23974908888339996 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4621114134788513 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 *jet/accuracy_1_of_3‚0.7975584864616394 +jet/accuracy_3_of_4‚0.04360226169228554 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9944405555725098 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3869580030441284 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_4_of_7‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14149919152259827 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5099726319313049 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.29139429330825806 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 jet/accuracy_5_of_7‚0 jet/accuracy_7_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3408038318157196 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9889018535614014 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9738159775733948 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.028777431696653366 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚0.5 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20783251523971558 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2885521650314331 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.012983269058167934 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9887150526046753 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.013631781563162804 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_7‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4464989900588989 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13234125077724457 particle/accuracy_2_of_4‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4020598828792572 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2747417390346527 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4019899070262909 *jet/accuracy_2_of_3‚0.2068169116973877 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1711014062166214 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.45064103603363037 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13261841237545013 particle/accuracy_5_of_6‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3430873155593872 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.25450190901756287 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12912198901176453 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0008818342466838658 particle/accuracy_3_of_7‚1 particle/accuracy_4_of_7‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9859843850135803 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3006761968135834 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.40414518117904663 jet/accuracy_5_of_6‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13060283660888672 +jet/accuracy_2_of_5‚0.40334102511405945 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 particle/accuracy_6_of_7‚1 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.41523176431655884 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.005931040737777948 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3950064182281494 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20228435099124908 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 *jet/accuracy_1_of_1‚0.4266137182712555 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07446808367967606 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0833333358168602 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.32863983511924744 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_3‚0.01576186530292034 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4018232524394989 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.18518516421318054 *"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5b–[Ü‘Ð ‚(:† ¢ÜèÔÁ¶ó­ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.584' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7168 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_úlÑ ‚(:á ¢ÜèԌ׬°" disk.dm-0.out‚30564.4208984375$ memory_percent‚89.65146541595459) proc.memory.availableMB‚ 1695.50390625 cpu‚24.883939486214 network.sent‚ 1.704018e+06 network.recv‚472312( disk./.usagePercent‚78.5837775337127 disk.sda.in‚0% disk.nvme0n1.out‚30564.4208984375 disk.dm-1.out‚ 41.01953125 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 4209.5390625# proc.memory.rssMB‚ 3026.73046875( proc.memory.percent‚18.4736967086792 proc.cpu.threads‚17% disk./.usageGB‚1170.9550743103027 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0.1875 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚30564.4208984375 disk.nvme0n1.in‚ 4209.5390625 disk.sda3.out‚ 41.01953125 disk.sda.out‚ 41.01953125 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 4209.5390625°èþ-“Ò ‚(:ˆ ±ÜèÔ’€Ã­ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.629' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3258 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…Ê1².Ó ‚(:ç ±ÜèÔ𺿯 disk.sda1.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚31853.5185546875 disk.nvme0n1.in‚ 4209.5390625# memory_percent‚92.4565315246582# proc.memory.rssMB‚ 2927.15234375 network.recv‚478922) disk./.usagePercent‚78.62013393167277$ disk./.usageGB‚1171.496810913086" disk.dm-0.out‚31853.5185546875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.865920066833496 network.sent‚ 1.828244e+06 disk.sda3.out‚ 41.79296875 disk.drbd1260.out‚ 0.19140625% disk.nvme0n1.out‚31853.5185546875 cpu‚24.886559141481943 disk.sda.out‚ 41.79296875 disk.dm-0.in‚ 4209.5390625 disk.sda3.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 4209.5390625' proc.memory.availableMB‚ 1235.921875 proEƸDc.cpu.threads‚17 disk.dm-1.out‚ 41.79296875 disk.sda.in‚0oY“Ô ‚(:ˆ ÀÜèÔù¿ê¯ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚14.792556762695314* gpu.0.memoryAllocatedBytes‚ 12706709504 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.081' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ†#þqÕ ‚(:æ ÀÜèÔ‘£Ú± disk.sda.out‚ 42.15234375 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 42.15234375 disk.dm-1.out‚ 42.15234375$ memory_percent‚92.45672225952148' proc.memory.availableMB‚ 1235.890625 proc.memory.rssMB‚ 2927.15625 network.recv‚483373) disk./.usagePercent‚78.65356621157676" disk.nvme0c0n1.in‚ 4209.5390625 disk.dm-1.in‚0 disk.drbd1260.in‚0* proc.memory.percent‚17.865943908691406 cpu‚24.888673899631154 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.in‚ 4209.5390625" disk.dm-0.out‚32204.4873046875 disk.drbd1260.out‚ 0.19140625' disk.nvme0c0n1.out‚32204.4873046875 disk.nvme0n1.in‚ 4209.5390625% disk.nvme0n1.out‚32204.4873046875 network.sent‚ 1.834192e+06% disk./.usageGB‚1171.99497604370122º“Î2Ö - _wandbruntime‚1390  _runtime‚13908‡›øÆ× À _runtime‚1390.820930753 _step‚99 & lr-AdamW/pg1‚0.0007922156917563833 & lr-AdamW/pg2‚0.0007922156917563833 trainer/global_step‚1249 % _timestamp‚1.7884892911804805e+09è…5ÊØ Ä trainer/global_step‚1249 % _timestamp‚1.7884892911804805e+09 _runtime‚1390.820930753 _step‚99 & lr-AdamW/pg1‚0.0007922156917563833 & lr-AdamW/pg2‚0.0007922156917563833c×u2Ù - _wandbruntime‚1390  _runtime‚1390þi!©Û'Ú ÕO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27739417 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚87510547772928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98507476036608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97163536348160 _step‚100 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚87510529769984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1280694129825 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27739417 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97163560288256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88795852369920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12569155 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86547689688069 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21918808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19603962 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚87510547772928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27762431 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209694720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88795852369920 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1281464778229 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚85266207870480 _runtime‚ 1390.82222405 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49681239 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1343729993728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1249½8¦âß'Û ÙO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1280694129825 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27739417 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97163560288256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88795852369920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12569155 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86547689688069 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21918808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19603962 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚87510547772928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27762431 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209694720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88795852369920 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1281464778229 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚85266207870480 _runtime‚ 1390.82222405 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49681239 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1343729993728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1249 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27739417 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚87510547772928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98507476036608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97163536348160 _step‚100 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚87510529769984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075d›„Õ¸2Ü - _wandbruntime‚1391  _runtime‚1391ñЬ!æ&Ý àM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49719850 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚87578247712256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19618300 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚85332274306240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234847232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21935680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1287100002816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1286327345664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1287100002816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚87578229709312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7039497 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97238884690944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27784168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1344756374528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚86614757263915 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚87578247712256 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88864557054976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1281695269703 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817£ØŽ›žÞ äM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1344967281664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27784168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UEDeviceStatsMonitor.on_train_batch_end/allocated_byt|ÿL%es.large_pool.peak‚ 10929229312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚85332291345600 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49696808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49719850 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21935680 trainer/global_step‚1249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49719850 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚87578247712256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19618300 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚85332274306240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234847232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21935680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1287100002816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1286327345664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1287100002816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚87578229709312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7039497 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97238884690944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27784168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1344756374528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚86614757263915 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚87578247712256 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88864557054976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1281695269703 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92231101036544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1395925705216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1336346944000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13051557 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚88569546697840 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22762508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51613841 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100930953487360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 trainer/global_step‚1299 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1336346944000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22762508 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22762506 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚92231891683328 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211791872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187851776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1441.292128794 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28851333 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28828319 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7296014 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚92231101036544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90895526736384 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1335574300160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51590825 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92231891683328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28851333 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20347054 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22762506 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90895526736384b¦"Dï ÞO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚89901082061177 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1335574300160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51590825 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92231891683328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28851333 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20347054 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22762506 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90895526736384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1335574300160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1330764714933 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13051555 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚Y'« 28113 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1331535363337 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92231101036544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1395925705216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1336346944000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13051557 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚88569546697840 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22762508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51613841 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100930953487360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 trainer/global_step‚1299 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1336346944000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22762508 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22762506 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚92231891683328 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211791872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187851776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1441.292128794 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28851333 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28828319 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7296014 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚92231101036544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90895526736384 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1336579045888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1337351703040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1397164907008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51652452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101006301830144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92300596381696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1332536503423 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1331765854811 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20361913 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚88635596094240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22779382 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28873070 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚1299 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101006325770240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13061203 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13061205 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22779382 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚89968149637023 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102403257927168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 $ _timestamp‚1.788489342902205e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90963244678656"î?í&ò çM EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1331765854811 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20361913 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚88635596094240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22779382 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28873070 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚1299 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101006325770240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13061203 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13061205 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22779382 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚89968149637023 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102403257927168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 $ _timestamp‚1.788489342902205e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90963244678656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20362438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51629410 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚89967361949051 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚105 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28873070 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28850030 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22779380 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1336579045888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1337351703040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1397164907008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51652452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101006301830144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92300596381696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1332536503423 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005inR¢2ó - _wandbruntime‚1442  _runtime‚1442@?’A_ô Ù 2loss/SRqqt4/assignment_loss‚1.4189576018120973 1loss/FBt4/assignment_loss‚0.18147881959144357 0loss/SRqqt2/detection_loss‚2.390947300376314 epoch‚1 0loss/FRt4/assignment_loss‚1.5124653780460358 /loss/FRt3/detection_loss‚2.0592985576198948 1loss/SRqqt4/detection_loss‚1.1984186608103975 /loss/FRt2/assignment_loss‚5.231478785646373 2loss/SRqqt3/assignment_loss‚1.9511964757243792 _runtime‚1442.546453967 _step‚106 /loss/FRt1/assignment_loss‚3.782365138593473 1loss/FBt1/assignment_loss‚0.40761541762577297 /loss/FBt1/detection_loss‚2.0963778770886936 % _timestamp‚1.7884893429060602e+09 .loss/FRt2/detection_loss‚2.401250189748304 1loss/SRqqt1/detection_loss‚2.3568172375361125 /loss/FBt3/detection_loss‚2.8353323340415955 /loss/FRt1/detection_loss‚2.6964739357170306 /loss/FRt4/detection_loss‚1.8154494116082787 .loss/FBt2/detection_loss‚2.288793236017227 /loss/FBt4/detection_loss‚0.7116278920217158 &loss/total_loss‚46.857113754758785 2loss/SRqqt2/assignment_loss‚2.0847740223010383 1loss/FBt2/assignment_loss‚0.36664065188961104 1loss/SRqqt3/detection_loss‚1.4432363279660543 trainer/global_step‚1299 2loss/SRqqt1/assignment_loss‚1.7296683877706527 1loss/FBt3/assignment_loss‚0.42061448039021343 0loss/FRt3/assignment_loss‚3.4758356348160775)5‰ cõ Ý 2loss/SRqqt1/assignment_loss‚1.7296683877706527 1loss/FBt3/assignment_loss‚0.42061448039021343 0loss/FRt3/assignment_loss‚3.4758356348160775 2loss/SRqqt4/assignment_loss‚1.4189576018120973 1loss/FBt4/assignment_loss‚0.18147881959144357 0loss/SRqqt2/detection_loss‚2.390947300376314 epoch‚1 0loss/FRt4/assignment_loss‚1.5124653780460358 /loss/FRt3/detection_loss‚2.0592985576198948 1loss/SRqqt4/detection_loss‚1.1984186608103975 /loss/FRt2/assignment_loss‚5.231478785646373 2loss/SRqqt3/assignment_loss‚1.9511964757243792 _runtime‚1442.546453967 _step‚106 /loss/FRt1/assignment_loss‚3.782365138593473 1loss/FBt1/assignment_loss‚0.40761541762577297 /loss/FBt1/detection_loss‚2.0963778770886936 % _timestamp‚1.7884893429060602e+09 .loss/FRt2/detection_loss‚2.401250189748304 1loss/SRqqt1/detection_loss‚2.3568172375361125 /loss/FBt3/detection_loss‚2.8353323340415955 /loss/FRt1/detection_loss‚2.6964739357170306 /loss/FRt4/detection_loss‚1.8154494116082787 .loss/FBt2/detection_loss‚2.288793236017227 /loss/FBt4/detection_loss‚0.7116278920217158 &loss/total_loss‚46.857113754758785 2loss/SRqqt2/assignment_loss‚2.0847740223010383 1loss/FBt2/assignment_loss‚0.36664065188961104 1loss/SRqqt3/detection_loss‚1.4432363279660543 trainer/global_step‚1299j«(5’ö ‚(:‡ ‹ÝèÔæ·§° gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.281' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0562 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~œÛi÷ ‚(:Þ ‹ÝèÔ®ÿ‰´ disk.sda3.out‚ 44.484375 disk.sda.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 4473.83984375# proc.memory.availableMB‚115.625 network.recv‚504853( disk./.usagePercent‚78.6808023411295 disk.dm-1.out‚ 44.484375 disk.sda1.out‚0 disk.sda2.out‚0# disk.dm-0.out‚33700.14501953125 cpu‚24.898917938711957 proc.cpu.threads‚17 network.sent‚ 1.96409e+06% disk./.usageGB‚1172.4008140563965 disk.sda3.in‚0 disk.drbd1260.out‚0.203125# disk.nvme0c0n1.in‚ 4473.83984375& disk.nvme0n1.out‚33700.14501953125* proc.memory.percent‚17.865943908691406 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 44.484375 disk.sda1.in‚0( disk.nvme0c0n1.out‚33700.14501953125 disk.dm-0.in‚ 4473.83984375$ memory_percent‚99.29428100585938 proc.memory.rssMB‚ 2927.15625´²3 ‘ø ‚(:† šÝèÔ’ÚÄ® gpu.0.gpu‚42.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.935' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.187 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ùœ)dqù ‚(:æ šÝèÔ¡àö³ network.sent‚ 2.071451e+06% disk./.usageGB‚1172.6518821716309 disk.sda.out‚ 44.86328125 disk.sda2.out‚0! disk.nvme0n1.in‚ 4473.83984375# disk.dm-0.out‚33910.76220703125* proc.memory.percent‚17.865943908691406 network.recv‚510296 disk.sda3.out‚ 44.86328125 disk.dm-1.out‚ 44.86328125 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 4473.83984375& disk.nvme0n1.out‚33910.76220703125 disk.dm-1.in‚0 disk.drbd1260.out‚0.203125 disk.dm-0.in‚ 4473.83984375$ memory_percent‚99.29275512695312 proc.memory.rssMB‚ 2927.15625 cpu‚24.901029970871058 proc.cpu.threads‚17) disk./.usagePercent‚78.69765173300307 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚33910.76220703125# proc.memory.availableMB‚115.875Ñ¢’ú ‚(:‡ ©ÝèÔ¿Õ„­ gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.063' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0126 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ†_Hiû ‚(:Þ ©ÝèÔÉôǯ' proc.memory.availableMB‚ 115.8671875* proc.memory.percent‚17.865943908691406 cpu‚24.9029203459106 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 45.2578125 disk.dm-1.out‚ 45.2578125 disk.sda.out‚ 45.2578125 disk.sda2.out‚0 proc.memory.rssMB‚ 2927.15625 network.recv‚515165 disk.dm-0.in‚ 4473.83984375 disk.dm-0.out‚34222.68359375 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 4473.83984375% disk.nvme0c0n1.out‚34222.68359375 network.sent‚ 2.077771e+06) disk./.usagePercent‚78.69779663326744% disk./.usageGB‚1172.6540412902832 disk.drbd1260.out‚ 0.20703125 disk.sda.in‚0! disk.nvme0n1.in‚ 4473.83984375# disk.nvme0n1.out‚34222.68359375$ memory_percent‚99.29280281066895 disk.drbd1260.in‚0 disk.sda1.in‚0E2ü - _wandbruntime‚1493  _runtime‚1493b/Çý Á & lr-AdamW/pg1‚0.0007919185466377877 & lr-AdamW/pg2‚0.0007919185466377877 trainer/global_step‚1349 % _timestamp‚1.7884893939232717e+09 _runtime‚1493.563634691 _step‚107¡öÐqþ UN6ÈÅ _step‚107 & lr-AdamW/pg1‚0.0007919185466377877 & lr-AdamW/pg2‚0.0007919185466377877 trainer/global_step‚1349 % _timestamp‚1.7884893939232717e+09 _runtime‚1493.563634691k1r\2ÿ - _wandbruntime‚1493  _runtime‚1493 kúðÜ'€ÖO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226471936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _step‚108 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95667140349952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1386598644224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1385826000384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23606208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23606206 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13533957 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29917221 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53523427 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29940235 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1448282069504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1380835300041 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚94280541705728 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1381605948445 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1385826000384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚94280541705728 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53546443 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚106146676636160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29940235 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7558062 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚93253686746521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21092533 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7558576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29917221 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚91872868485840 trainer/global_step‚1349 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13533955 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1448079537664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94280523702784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1386598644224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23606206 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106146450164224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202531840 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚93254474434285 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91872851446480 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95666349703168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104698394566656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104698370626560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94280523702784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21092017 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 # _timestamp‚1.78848939392446e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _runtime‚ 1493.5649825 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23606208 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚95666349703168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1385826000384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚94280541705728 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53546443 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚106146676636160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29940235 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7558062 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚93253686746521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21092533 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7558576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29917221 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚91872868485840 trainer/global_step‚1349 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13533955 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1448079537664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94280523702784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1386598644224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23606206 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106146450164224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202531840 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚93254474434285 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91872851446480 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95666349703168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104698394566656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104698370626560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94280523702784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21092017 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 # _timestamp‚1.78848939392446e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _runtime‚ 1493.5649825l™wf2‚- _wandbruntime‚1494  _runtime‚1494C°ê&ƒäM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1449141576192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95735845048320 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7563315 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1349 _runtime‚1494.449795767 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23623080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106222860545536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884893948085766e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53585054 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53585054 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29961972 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7563819 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53562012 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95735845048320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94348241645056 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21106918 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23623080 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189935616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1387603403264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23623082 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1387603403264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91938934921600 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29961972 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1386830746112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1449331511808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚94348223642112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29938932 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1386830746112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104773742909440¬B ý„èM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1387603403264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23623082 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1387603403264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91938934921600 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29961972 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1386830746112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1449331511808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚94348223642112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29938932 GADeviceStatsMonitor.ï˜OÙñon_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1386830746112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104773742909440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1449141576192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95735845048320 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7563315 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1349 _runtime‚1494.449795767 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23623080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106222860545536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884893948085766e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53585054 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53585054 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29961972 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7563819 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53562012 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95735845048320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94348241645056 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21106918 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23623080 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189935616míX|2…- _wandbruntime‚1494  _runtime‚1494?ÌöÊW†Ñ 1loss/SRqqt2/assignment_loss‚2.335153943962521 1loss/SRqqt1/detection_loss‚2.2024217320651545 1loss/SRqqt2/detection_loss‚2.6292323999934726 .loss/FBt3/detection_loss‚3.390138347943624 &loss/total_loss‚51.799117262134985 epoch‚1 _step‚110 /loss/FRt2/detection_loss‚2.7345206181208295 trainer/global_step‚1349 /loss/FRt3/assignment_loss‚3.933029948619374 /loss/FBt1/assignment_loss‚1.386665366590023 0loss/FBt2/assignment_loss‚0.6770346446169747 .loss/FRt1/detection_loss‚2.557829534014066 1loss/SRqqt3/detection_loss‚1.5892004968805447 /loss/FBt1/detection_loss‚2.9891629107296467 % _timestamp‚1.7884893948105187e+09 _runtime‚1494.451209932 0loss/FBt4/assignment_loss‚0.2400770820287867 1loss/SRqqt4/detection_loss‚1.0487332477592506 .loss/FBt2/detection_loss‚2.799014515346951 /loss/FRt4/assignment_loss‚1.890011077457004 1loss/SRqqt1/assignment_loss‚1.822422247834322 /loss/FRt3/detection_loss‚2.2492346554471734 /loss/FRt2/assignment_loss‚4.795748496055603 2loss/SRqqt3/assignment_loss‚1.8814944526077144 /loss/FBt4/detection_loss‚0.6944524129773864 0loss/FRt1/assignment_loss‚3.3852192418915883 2loss/SRqqt4/assignment_loss‚0.9496300043681494 0loss/FBt3/assignment_loss‚1.4399658677478062 /loss/FRt4/detection_loss‚2.1787240170770223•Çt[‡Õ /loss/FRt2/assignment_loss‚4.795748496055603 2loss/SRqqt3/assignment_loss‚1.8814944526077144 /loss/FBt4/detection_loss‚0.6944524129773864 0loss/FRt1/assignment_loss‚3.3852192418915883 2loss/SRqqt4/assignment_loss‚0.9496300043681494 0loss/FBt3/assignment_loss‚1.4399658677478062 /loss/FRt4/detection_loss‚2.1787240170770223 1loss/SRqqt2/assignment_loss‚2.335153943962521 1loss/SRqqt1/detection_loss‚2.2024217320651545 1loss/SRqqt2/detection_loss‚2.6292323999934726 .loss/FBt3/detection_loss‚3.390138347943624 &loss/total_loss‚51.799117262134985 epoch‚1 _step‚110 /loss/FRt2/detection_loss‚2.7345206181208295 trainer/global_step‚1349 /loss/FRt3/assignment_loss‚3.933029948619374 /loss/FBt1/assignment_loss‚1.386665366590023 0loss/FBt2/assignment_loss‚0.6770346446169747 .loss/FRt1/detection_loss‚2.557829534014066 1loss/SRqqt3/detection_loss‚1.5892004968805447 /loss/FBt1/detection_loss‚2.9891629107296467 % _timestamp‚1.7884893948105187e+09 _runtime‚1494.451209932 0loss/FBt4/assignment_loss‚0.2400770820287867 1loss/SRqqt4/detection_loss‚1.0487332477592506 .loss/FBt2/detection_loss‚2.799014515346951 /loss/FRt4/assignment_loss‚1.890011077457004 1loss/SRqqt1/assignment_loss‚1.822422247834322 /loss/FRt3/detection_loss‚2.2492346554471734ne' ˆ‚(:’ ¸ÝèÔò÷èº gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.242' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.848399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓÒàj‰‚(:ß ¸ÝèÔ“¯­½ disk.dm-0.in‚ 4473.83984375 disk.dm-0.out‚34743.01171875 disk.sda3.out‚ 45.72265625 disk.dm-1.in‚0& proc.memory.availableMB‚ 115.546875 proc.memory.rssMB‚ 2927.15625 network.sent‚ 2.08372e+06 disk.drbd1260.out‚ 0.20703125% disk.nvme0c0n1.out‚34743.01171875 disk.sda3.in‚0* proc.memory.percent‚17.865943908691406 cpu‚24.90497768363022 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 4473.83984375! disk.nvme0n1.in‚ 4473.83984375# disk.nvme0n1.out‚34743.01171875 network.recv‚519678% disk./.usageGB‚1172.6618118286133 disk.sda.out‚ 45.72265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 45.72265625$ memory_percent‚99.29475784301758( disk./.usagePercent‚78.6983181206147 disk.drbd1260.in‚0X4"Š‚(:’ ÇÝèÔõ”Ö­ gpu.0.gpu‚38.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.341' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.068199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03Gòn‹‚(:ã ÇÝèÔªÏƯ network.sent‚ 2.191241e+06) disk./.usagePercent‚78.69831556053936( disk.nvme0c0n1.out‚34895.07861328125 disk.sda3.in‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 115.3828125% disk./.usageGB‚1172.6617736816406 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚34895.07861328125 disk.dm-0.in‚ 4617.640625 disk.drbd1260.out‚ 0.2109375 proc.memory.rssMB‚ 2927.15625 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 4617.640625# disk.dm-0.out‚34895.07861328125 disk.drbd1260.in‚0 disk.sda.out‚ 46.26171875 cpu‚24.90678378464064 proc.cpu.threads‚17 network.recv‚525382 disk.nvme0n1.in‚ 4617.640625 disk.sda3.out‚ 46.26171875 disk.dm-1.out‚ 46.26171875 disk.sda.in‚0# memory_percent‚99.2957592010498* proc.memory.percent‚17.865943908691406s›Œ‚(:… ÖÝèỒư gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚127.32' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.464 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MOņq‚(:æ ÖÝèÔÖ›Ô² disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 4617.640625$ memory_percent‚99.29425716400146 network.sent‚ 2.196423e+06 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚35003.55126953125( proc.memory.availableMB‚ 115.62890625 proc.memory.rssMB‚ 2927.15625* proc.memory.percent‚17.865943908691406% disk./.usageGB‚1173.1520614624023 disk.nvme0n1.in‚ 4617.640625 disk.sda3.out‚ 46.68359375 disk.dm-1.out‚ 46.68359375 disk.sda.in‚0 cpu‚24.908640394632183 proc.cpu.threads‚17) disk./.usagePercent‚78.73121918488512& disk.nvme0n1.out‚35003.55126953125 disk.dm-0.in‚ 4617.640625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.2109375 disk.sda2.out‚0 network.recv‚528937# disk.dm-0.out‚35003.55126953125 disk.sda.out‚ 46.68359375Ž„¶ñ2Ž- _wandbruntime‚1542  _runtime‚1542÷)ØÏÇÁ % _timestamp‚1.7884894431446033e+09 _runtime‚1542.784974809 _step‚111 & lr-AdamW/pg1‚0.0007916462288916086 & lr-AdamW/pg2‚0.0007916462288916086 trainer/global_step‚1399Õj‘•ËÅ & lr-AdamW/pg1‚0.0007916462288916086 & lr-AdamW/pg2‚0.0007916462288916086 trainer/global_step‚1399 % _timestamp‚1.7884894431446033e+09 _runtime‚1542.784974809 _step‚111oô„Ò2‘- _wandbruntime‚1542  _runtime‚1542¡’£¯à'’ÚO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99102389016576 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99101598369792 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55456029 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1436850344448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24449908 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55479045 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95176190273840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1436850344448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚109966301884416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96607866807393 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31006123 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97665520669184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7820514 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99101598369792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1436077700608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1500490178560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1430905885149 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 trainer/global_step‚1399 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108465811705856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1436077700608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 % _timestamp‚1.7884894431456468e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31029137 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96607079119629 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24449906 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21836326 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108465787765760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21836871 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚109966077509632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99101598369792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1436077700608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1500490178560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1430905885149 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 trainer/global_step‚1399 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108465811705856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1436077700608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 % _timestamp‚1.7884894431456468e+09 E?DeviceStatsMonitor.on_train_batch_Ò ø start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31029137 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96607079119629 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24449906 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21836326 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108465787765760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21836871 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚109966077509632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55517656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1432677673639 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110042700697600 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14026003 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97733220608512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95242239670240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24466780 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110042486821888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96674146695267 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1501350713344 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚113 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55494614 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99170303054848 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21851219 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14026005 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1437855103488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189935616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24466782 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97733238611456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1437082446336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1431907025027 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584Ö¬î&–èM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24466780 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110042486821888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96674146695267 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1501350713344 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚113 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55494614 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99170303054848 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21851219 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14026005 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1437855103488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189935616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24466782 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97733238611456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1437082446336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1431907025027 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7825749 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31027834 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31050874 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1399 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21851754 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 % _timestamp‚1.7884894440419827e+09 _runtime‚1543.683059361 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24466782 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108541160048640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24466780 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1437855103488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55517656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1432677673639 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110042700697600 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14026003 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97733220608512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95242239670240q.ÖU2—- _wandbruntime‚1543  _runtime‚1543bFsŽY˜Ó $ _timestamp‚1.788489444044759e+09 2loss/SRqqt2/assignment_loss‚2.0445130704080356 1loss/SRqqt3/assignment_loss‚1.858468669029523 0loss/FBt1/assignment_loss‚0.3102292207671174 2loss/FBt3/assignment_loss‚0.004466886499025284 1loss/FBt4/assignment_loss‚0.18228886707278974 /loss/FRt2/detection_loss‚2.6316136522539733 1loss/SRqqt4/detection_loss‚0.9654583464775767 2loss/SRqqt1/assignment_loss‚1.7025779329049282 /loss/FRt1/detection_loss‚2.7935760789795925 0loss/SRqqt1/detection_loss‚2.414458317634387 1loss/SRqqt2/detection_loss‚2.5040383812259224 1loss/SRqqt3/detection_loss‚1.9539115289036109 0loss/FRt1/assignment_loss‚4.1643710826572615 /loss/FRt2/assignment_loss‚4.422310196120163 1loss/SRqqt4/assignment_loss‚1.246307238670332 /loss/FBt4/detection_loss‚0.5495563548920303 -loss/FBt3/detection_loss‚3.41665324143001 .loss/FRt3/detection_loss‚2.178173982139145 .loss/FBt2/detection_loss‚3.937278767426809 _runtime‚1543.684966696 /loss/FBt1/detection_loss‚3.6815832257270813 %loss/total_loss‚49.40422201589864 trainer/global_step‚1399 0loss/FRt3/assignment_loss‚3.1906904067311967 1loss/FBt2/assignment_loss‚0.01606416377694586 .loss/FRt4/detection_loss‚1.602903844981358 0loss/FRt4/assignment_loss‚1.6327285591898293 epoch‚1 _step‚114স]™× 0loss/FRt3/assignment_loss‚3.1906904067311967 1loss/FBt2/assignment_loss‚0.01606416377694586 .loss/FRt4/detection_loss‚1.602903844981358 0loss/FRt4/assignment_loss‚1.6327285591898293 epoch‚1 _step‚114 $ _timestamp‚1.788489444044759e+09 2loss/SRqqt2/assignment_loss‚2.0445130704080356 1loss/SRqqt3/assignment_loss‚1.858468669029523 0loss/FBt1/assignment_loss‚0.3102292207671174 2loss/FBt3/assignment_loss‚0.004466886499025284 1loss/FBt4/assignment_loss‚0.18228886707278974 /loss/FRt2/detection_loss‚2.6316136522539733 1loss/SRqqt4/detection_loss‚0.9654583464775767 2loss/SRqqt1/assignment_loss‚1.7025779329049282 /loss/FRt1/detection_loss‚2.7935760789795925 0loss/SRqqt1/detection_loss‚2.414458317634387 1loss/SRqqt2/detection_loss‚2.5040383812259224 1loss/SRqqt3/detection_loss‚1.9539115289036109 0loss/FRt1/assignment_loss‚4.1643710826572615 /loss/FRt2/assignment_loss‚4.422310196120163 1loss/SRqqt4/assignment_loss‚1.246307238670332 /loss/FBt4/detection_loss‚0.5495563548920303 -loss/FBt3/detection_loss‚3.41665324143001 .loss/FRt3/detection_loss‚2.178173982139145 .loss/FBt2/detection_loss‚3.937278767426809 _runtime‚1543.684966696 /loss/FBt1/detection_loss‚3.6815832257270813 %loss/total_loss‚49.40422201589864 trainer/global_step‚1399rÉOò]’š‚(:‡ åÝèԣĚ¯ gpu.0.gpu‚45.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.377' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09©—p›‚(:å åÝèÔ©³ð°! disk.nvme0c0n1.in‚ 4617.640625& disk.nvme0n1.out‚35406.95263671875 disk.dm-0.in‚ 4617.640625$ memory_percent‚99.29611682891846 disk.dm-1.out‚ 47.06640625 disk.sda.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 2927.15625 cpu‚24.91046159742208 proc.cpu.threads‚17 network.recv‚533560 disk.sda1.out‚0( disk.nvme0c0n1.out‚35406.95263671875 disk.nvme0n1.in‚ 4617.640625( proc.memory.availableMB‚ 115.32421875* proc.memory.percent‚17.865943908691406) disk./.usagePercent‚78.66695514960999 disk.sda3.in‚0 disk.sda3.out‚ 47.06640625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.21484375 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚35406.95263671875 network.sent‚ 2.215552e+06$ disk./.usageGB‚1172.194480895996 disk.dm-1.in‚0 disk.sda.out‚ 47.06640625‘†#œ‚(:’ ôÝèÔœè­ gpu.0.gpu‚48.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.564' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.312799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È“_ÿp‚(:å ôÝèÔ´øÛ® disk.sda2.in‚0( disk.nvme0c0n1.out‚35882.70263671875 disk.dm-0.in‚ 4617.640625 proc.memory.rssMB‚ 2927.15625 proc.cpu.threads‚17) disk./.usagePercent‚78.67689028999878 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.21484375 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚35882.70263671875# disk.dm-0.out‚35882.70263671875* proc.memory.percent‚17.865943908691406 disk.dm-1.out‚ 47.58203125 disk.drbd1260.in‚0 disk.sda.out‚ 47.58203125 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 4617.640625 disk.nvme0n1.in‚ 4617.640625# memory_percent‚99.2971658706665( proc.memory.availableMB‚ 115.15234375 network.sent‚ 2.310929e+06 network.recv‚539728 disk.sda3.in‚0 disk.sda3.out‚ 47.58203125 disk.sda.in‚0 cpu‚24.91211075269815% disk./.usageGB‚1172.3425216674805lQPž‚(:’ ƒÞèÔéÿ‘° gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.454' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.890800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Ô1kŸ‚(:à ƒÞèÔ…´ disk.nvme0n1.in‚ 4716.828125 disk.drbd1260.out‚0.21875$ memory_percent‚99.29587841033936 proc.memory.rssMB‚ 2927.15625 proc.cpu.threads‚17) disk./.usagePercent‚78.69216677158197% disk.nvme0n1.out‚36172.7939453125" disk.dm-0.out‚36172.7939453125 disk.sda3.in‚0( proc.memory.availableMB‚ 115.36328125 disk.dm-1.out‚ 47.91796875 network.sent‚ 2.317809e+06 network.recv‚542865% disk./.usageGB‚1172.5701522827148 disk.dm-0.in‚ 4716.828125 disk.sda3.out‚ 47.91796875 disk.drbd1260.in‚0 disk.sda.out‚ 47.91796875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.865943908691406 cpu‚24.91394830161083! disk.nvme0c0n1.in‚ 4716.828125' disk.nvme0c0n1.out‚36172.7939453125Eïx5’ ‚(:‡ ’ÞèÔ€¨Á® gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.784' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅÍÈÙa¡‚(:Ö ’ÞèÔÀÙ¥² cpu‚24.915825929942613 proc.cpu.threads‚17 network.recv‚546189! disk.nvme0c0n1.in‚ 4716.828125' disk.nvme0c0n1.out‚36557.6376953125% disk.nvme0n1.out‚36557.6376953125 disk.dm-0.in‚ 4716.828125 disk.sda3.out‚48.28125$ memory_percent‚99.29590225219727* proc.memory.percent‚17.865943908691406 disk.nvme0n1.in‚ 4716.828125" disk.dm-0.out‚36557.6376953125 disk.sda3.in‚0 disk.dm-1.out‚48.28125 disk.drbd1260.in‚0 disk.sda.in‚0 proc.memory.rssMB‚ 2927.15625 network.sent‚ 2.320218e+06) disk./.usagePercent‚78.69367107185316% disk./.usageGB‚1172.5925674438477 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0.21875 disk.sda2.in‚0 disk.sda.out‚48.28125& proc.memory.availableMB‚ 115.359375j³å2¢- _wandbruntime‚1593  _runtime‚1593Òö0Ç£Á _step‚115 & lr-AdamW/pg1‚0.0007912914842918355 & lr-AdamW/pg2‚0.0007912914842918355 trainer/global_step‚1449 % _timestamp‚1.7884894939884586e+09 _runtime‚1593.629004719X•/5ˤŠ_runtime‚1593.629004719 _step‚115 & lr-AdamW/pg1‚0.0007912914842918355 & lr-AdamW/pg2‚0.0007912914842918355 trainer/global_step‚1449 % _timestamp‚1.7884894939884586e+09s‘ÛÙÿ2¥- _wandbruntime‚1593  _runtime‚1593¥~YÒp¦âO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚102536847036416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57388631 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚101050535638528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14498757 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚1593.630245999 VGªÙÓkx"DeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884894939896827e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _step‚116 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚101050517635584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224374784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25293608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32095025 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1486329400832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98479495022480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102536847036416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚99961259180501 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98479512061840 trainer/global_step‚1449 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200434688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚101050535638528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101050517635584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1480976470257 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32118039 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99960471492737 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25293608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57411647 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1552657504768 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1487102044672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32118039 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8083884 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1552857939456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚102537637683200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32095025 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1481747118661 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25293606 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102537637683200 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8084409 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚113786086784512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112233204904960 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25293606 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22583166 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1486329400832 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22582639 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112233228845056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Û‚bÙì'§æO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25293606 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22583166 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1486329400832 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22582639 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112233228845056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚102536847036416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57388631 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚101050535638528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14498757 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚1593.630245999 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884894939896827e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _step‚116 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚101050517635584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224374784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25293608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32095025 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1486329400832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98479495022480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102536847036416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚99961259180501 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98479512061840 trainer/global_step‚1449 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200434688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚101050535638528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101050517635584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1480976470257 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32118039 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99960471492737 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25293608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57411647 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1552657504768 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1487102044672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32118039 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8083884 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1552857939456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚102537637683200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32095025 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1481747118661 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25293606 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102537637683200 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8084409 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚113786086784512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112233204904960t#µþ2¨- _wandbruntime‚1594  _runtime‚1594½!Æô&©îM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101118217574912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚1594.497644117 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101118217574912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204615680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25310480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚1449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8089154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25310482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14508405 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 _step‚117 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57427216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32139776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22597557 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1487334146560 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57450258 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22598072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1553899806208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1553695190528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102605551721472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98545578497600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101118235577856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32139776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112308577187840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101118235577856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100028326756347 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1481977610135 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1482748258747 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EÖÁSªòM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 _step‚117 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57427216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32139776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22597557 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1487334146560 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57450258 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22598072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1553899806208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1553695190528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102605551721472 4.DeviceStatsMonitor.on_train_,BŽÝ¥batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98545578497600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101118235577856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32139776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112308577187840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101118235577856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100028326756347 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1481977610135 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1482748258747 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25310480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚1449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8089154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25310482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14508405 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101782816810480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14981157 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117605620900352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116000645984256 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1536581101056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26137306 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33206941 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105972886349824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14981155 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105972095703040 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8347399 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _step‚120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1537353744896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26137308 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105972095703040 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8347399 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _step‚120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1537353744896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26137308 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101782816810480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992xHªú2º- _wandbruntime‚1643  _runtime‚1643/7Ñõ&»ïM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1605817167360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59382860 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33228678 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14990803 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚106041591048192 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚104503214541312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1537585846784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚103380931441483 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1532048195243 $ _timestamp‚1.788489544145649e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26154182 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33205638 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26154180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33205638 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33228678 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59359818 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26154182 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26154180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23343962 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1532818843855 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1538358503936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101848900285600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8352633 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117682005624320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚104503232544256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8353157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1606011297280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23343436 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59359818 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚106041591048192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚103381719129455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1643.786593699 _step‚121'ä÷Eù&¼óM EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59382860 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33228678 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14990803 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚106041591048192 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚104503214541312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1537585846784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚103380931441483 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1532048195243 $ _timestamp‚1.788489544145649e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26154182 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33205638 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26154180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33205638 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33228678 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59359818 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26154182 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26154180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23343962 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1532818843855 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1538358503936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101848900285600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8352633 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117682005624320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚104503232544256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8353157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1606011297280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23343436 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59359818 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚106041591048192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚103381719129455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1643.786593699 _step‚121 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1605817167360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040yõ‰Àà2½- _wandbruntime‚1643  _runtime‚1643‘±K¢[¾Õ 2loss/SRqqt2/assignment_loss‚2.5707797650247812 1loss/SRqqt3/assignment_loss‚1.455157305190309 /loss/FRt1/detection_loss‚3.0593237533019138 /loss/FRt4/detection_loss‚1.5486005579806723 1loss/SRqqt3/detection_loss‚1.6457571050182718 .loss/FBt2/detection_loss‚2.508004221651289 &loss/total_loss‚45.933629457926216 % _timestamp‚1.7884895441481497e+09 0loss/FRt1/assignment_loss‚3.9554200906019945 /loss/FRt2/detection_loss‚2.4936531105557003 _step‚122 0loss/FRt3/assignment_loss‚2.3346062404353445 1loss/FBt2/assignment_loss‚0.18123246211244906 0loss/FBt1/assignment_loss‚0.5073976905938858 ,loss/FBt1/detection_loss‚2.5628622174263 .loss/FRt3/detection_loss‚2.453272419731791 /loss/FBt3/detection_loss‚3.4634363412857057 epoch‚1 2loss/SRqqt4/assignment_loss‚0.7434812634725307 1loss/FBt3/assignment_loss‚0.02088557560928166 0loss/SRqqt1/detection_loss‚2.510356266157968 1loss/SRqqt2/detection_loss‚2.3718828298151493 0loss/FRt4/assignment_loss‚1.6589184583358045 2loss/SRqqt1/assignment_loss‚1.4289344786533287 1loss/FBt4/assignment_loss‚0.15269186055806838 1loss/SRqqt4/detection_loss‚1.1864299369273617 /loss/FBt4/detection_loss‚0.6172066518202363 trainer/global_step‚1499 _runtime‚1643.788308212 /loss/FRt2/assignment_loss‚4.503338855666083¸*†_¿Ù 2loss/SRqqt2/assignment_loss‚2.5707797650247812 1loss/SRqqt3/assignment_loss‚1.455157305190309 /loss/FRt1/detection_loss‚3.0593237533019138 /loss/FRt4/detection_loss‚1.5486005579806723 1loss/SRqqt3/detection_loss‚1.6457571050182718 .loss/FBt2/detection_loss‚2.508004221651289 &loss/total_loss‚45.933629457926216 % _timestamp‚1.7884895441481497e+09 0loss/FRt1/assignment_loss‚3.9554200906019945 /loss/FRt2/detection_loss‚2.4936531105557003 _step‚122 0loss/FRt3/assignment_loss‚2.3346062404353445 1loss/FBt2/assignment_loss‚0.18123246211244906 0loss/FBt1/assignment_loss‚0.5073976905938858 ,loss/FBt1/detection_loss‚2.5628622174263 .loss/FRt3/detection_loss‚2.453272419731791 /loss/FBt3/detection_loss‚3.4634363412857057 epoch‚1 2loss/SRqqt4/assignment_loss‚0.7434812634725307 1loss/FBt3/assignment_loss‚0.02088557560928166 0loss/SRqqt1/detection_loss‚2.510356266157968 1loss/SRqqt2/detection_loss‚2.3718828298151493 0loss/FRt4/assignment_loss‚1.6589184583358045 2loss/SRqqt1/assignment_loss‚1.4289344786533287 1loss/FBt4/assignment_loss‚0.15269186055806838 1loss/SRqqt4/detection_loss‚1.1864299369273617 /loss/FBt4/detection_loss‚0.6172066518202363 trainer/global_step‚1499 _runtime‚1643.788308212 /loss/FRt2/assignment_loss‚4.503338855666083zLIj¢’À‚(:‡ ÎÞèÔëéç® gpu.0.gpu‚18.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.377' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mp˨eÁ‚(:Ú ÎÞèÔèÿƒ±% disk./.usageGB‚1173.1573753356934 disk.dm-0.in‚ 4848.8359375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 4848.8359375$ memory_percent‚99.29673671722412 network.sent‚ 2.444854e+06 network.recv‚564499 disk.sda3.out‚ 49.85546875 disk.sda.out‚ 49.85546875 disk.sda1.in‚0# disk.nvme0c0n1.out‚ 37814.140625( proc.memory.availableMB‚ 115.22265625 cpu‚24.922104047725334) disk./.usagePercent‚78.73157580338031 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 4848.8359375 proc.memory.rssMB‚ 2927.15625* proc.memory.percent‚17.865943908691406 disk.dm-0.out‚ 37814.140625 disk.dm-1.out‚ 49.85546875 disk.drbd1260.out‚ 0.2265625 disk.sda.in‚0 disk.sda2.out‚0! disk.nvme0n1.out‚ 37814.140625 proc.cpu.threads‚17=uœ¡’‚(:‡ ÝÞèÔå«ñ® gpu.0.gpu‚48.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.107' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸Z8¤tÂ(:é ÝÞèÔêê²² network.sent‚ 2.553635e+06 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 50.40234375 disk.sda1.out‚0 disk.sda2.in‚0 network.recv‚571056) disk./.usagePercent‚78.73187277212004& disk.nvme0n1.out‚38199.36669921875 proc.memory.rssMB‚ 2927.15625 cpu‚24.923879039089964 proc.cpu.threads‚17% disk./.usageGB‚1173.1618003845215" disk.nvme0c0n1.in‚ 4848.8359375( disk.nvme0c0n1.out‚38199.36669921875 disk.dm-0.in‚ 4848.8359375 disk.sda3.in‚0 disk.drbd1260.out‚ 0.23046875 disk.sda.in‚0$ memory_percent‚99.29709434509277 disk.nvme0n1.in‚ 4848.8359375# disk.dm-0.out‚38199.36669921875 disk.sda3.out‚ 50.40234375 disk.dm-1.out‚ 50.40234375 disk.sda1.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 115.1640625* proc.memory.percent‚17.865943908691406»ð§²Ä‚(:’ ìÞèÔ›ÒÌ° gpu.0.gpu‚12.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.062200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡›&oÅ‚(:ä ìÞèÔ¼šÔ² disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 4848.8359375# disk.dm-0.out‚38299.19873046875$ memory_percent‚99.29864406585693 network.recv‚575211$ disk./.usageGB‚1173.161808013916 disk.sda3.out‚ 50.8984375 disk.dm-1.out‚ 50.8984375 disk.sda2.in‚0 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.23046875 disk.sda.out‚ 50.8984375( disk.nvme0c0n1.out‚38299.19873046875 disk.nvme0n1.in‚ 4848.8359375& disk.nvme0n1.out‚38299.19873046875( proc.memory.availableMB‚ 114.91015625 proc.memory.rssMB‚ 2927.15625 network.sent‚ 2.559686e+06( disk./.usagePercent‚78.7318732841351 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 4848.8359375* proc.memory.percent‚17.865943908691406 cpu‚24.92537266947153 disk.dm-1.in‚0 q|y2Æ- _wandbruntime‚1692  _runtime‚1692w´CkÅÇ¿ & lr-AdamW/pg1‚0.0007903350274067056 & lr-AdamW/pg2‚0.0007903350274067056 trainer/global_step‚1549 $ _timestamp‚1.788489592806223e+09 _runtime‚ 1692.44671031 _step‚123]÷­ÉÈà & lr-AdamW/pg1‚0.0007903350274067056 & lr-AdamW/pg2‚0.0007903350274067056 trainer/global_step‚1549 $ _timestamp‚1.788489592806223e+09 _runtime‚ 1692.44671031 _step‚123{EÝñÂ2É- _wandbruntime‚1692  _runtime‚1692o¢T'¨ÊæO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26981008 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24075462 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109408135016448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884895928075194e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1581117640473 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26981006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202531840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107820529571328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8611905 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61276851 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106668043926717 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109407344369664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121425035960832 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚105086155637840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109407344369664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121425035960832 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚105086155637840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24075462 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109408135016448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884895928075194e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1581117640473 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26981006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202531840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107820529571328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8611905 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61276851 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106668043926717 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24090319 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚105152222073600 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109476839714816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119843387526144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 _step‚125 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61315462 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61292420 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109476839714816 % _timestamp‚1.7884895936997278e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107888211507712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26997882 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26997882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15473203 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119843411466240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34294540 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1582889428963 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24089789 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196227072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26997880 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8616586 trainer/global_step‚1549 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1588610204160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34317580 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1588610204160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1587837547008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1582118780351 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1693.341428001 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107888211507712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61315462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109476049054720 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¡®6ÎôM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119843387526144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 _step‚125 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61315462 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61292420 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109476839714816 % _timestamp‚1.7884895936997278e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107888211507712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26997882 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26997882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15473203 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119843411466240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34294540 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1582889428963 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24089789 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196227072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26997880 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8616586 trainer/global_step‚1549 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1588610204160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3431‰¬Žy 7580 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1588610204160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1587837547008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1582118780351 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1693.341428001 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107888211507712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61315462 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109476049054720 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61292420 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26997880 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8617114 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121501652794880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24090319 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚105152222073600 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109476839714816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2}}8£E2Ï- _wandbruntime‚1693  _runtime‚1693:t-ßXÐÒ /loss/FRt2/assignment_loss‚4.145356492084615 1loss/SRqqt3/detection_loss‚1.7047833025455474 2loss/SRqqt1/assignment_loss‚2.2365029404560723 1loss/SRqqt2/assignment_loss‚2.409284634836789 %loss/total_loss‚48.21687588309317 _runtime‚1693.343647425 0loss/FBt3/assignment_loss‚0.6796546367847642 /loss/FRt3/assignment_loss‚3.527494152761855 0loss/FRt4/assignment_loss‚1.4852225577577631 2loss/SRqqt3/assignment_loss‚1.8229706678849955 1loss/FBt2/assignment_loss‚0.20233834242786874 1loss/SRqqt1/detection_loss‚2.6283487611346774 0loss/SRqqt4/detection_loss‚1.164995213287572 .loss/FBt1/detection_loss‚3.321300254927741 0loss/SRqqt2/detection_loss‚2.864140212535858 .loss/FBt2/detection_loss‚2.524322824044661 /loss/FBt4/detection_loss‚0.5193751162184543 epoch‚1 1loss/FBt4/assignment_loss‚0.19955820583013056 /loss/FRt2/detection_loss‚2.4298978114829346 2loss/SRqqt4/assignment_loss‚0.9964493840932847 0loss/FBt1/assignment_loss‚0.1700930724017932 % _timestamp‚1.7884895937032974e+09 .loss/FRt3/detection_loss‚2.111803078988813 /loss/FRt4/detection_loss‚1.6979023467036003 trainer/global_step‚1549 /loss/FRt1/assignment_loss‚3.956013878186544 /loss/FRt1/detection_loss‚2.7426139947139854 /loss/FBt3/detection_loss‚2.6764540010028415 _step‚126ØŒ\ÑÖ .loss/FRt3/detection_loss‚2.111803078988813 /loss/FRt4/detection_loss‚1.6979023467036003 trainer/global_step‚1549 /loss/FRt1/assignment_loss‚3.956013878186544 /loss/FRt1/detection_loss‚2.7426139947139854 /loss/FBt3/detection_loss‚2.6764540010028415 _step‚126 /loss/FRt2/assignment_loss‚4.145356492084615 1loss/SRqqt3/detection_loss‚1.7047833025455474 2loss/SRqqt1/assignment_loss‚2.2365029404560723 1loss/SRqqt2/assignment_loss‚2.409284634836789 %loss/total_loss‚48.21687588309317 _runtime‚1693.343647425 0loss/FBt3/assignment_loss‚0.6796546367847642 /loss/FRt3/assignment_loss‚3.527494152761855 0loss/FRt4/assignment_loss‚1.4852225577577631 2loss/SRqqt3/assignment_loss‚1.8229706678849955 1loss/FBt2/assignment_loss‚0.20233834242786874 1loss/SRqqt1/detection_loss‚2.6283487611346774 0loss/SRqqt4/detection_loss‚1.164995213287572 .loss/FBt1/detection_loss‚3.321300254927741 0loss/SRqqt2/detection_loss‚2.864140212535858 .loss/FBt2/detection_loss‚2.524322824044661 /loss/FBt4/detection_loss‚0.5193751162184543 epoch‚1 1loss/FBt4/assignment_loss‚0.19955820583013056 /loss/FRt2/detection_loss‚2.4298978114829346 2loss/SRqqt4/assignment_loss‚0.9964493840932847 0loss/FBt1/assignment_loss‚0.1700930724017932 % _timestamp‚1.7884895937032974e+09~ƒC±¿’Ò‚(:‡ ûÞèÔ‰ßѯ gpu.0.gpu‚48.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.004' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bäònÓ‚(:ã ûÞèÔ㇟°$ memory_percent‚99.29144382476807 proc.memory.rssMB‚ 2927.15625* proc.memory.percent‚17.865943908691406 proc.cpu.threads‚17$ disk./.usageGB‚1173.224208831787 disk.sda3.in‚0 disk.drbd1260.out‚0.234375( proc.memory.availableMB‚ 116.08984375 network.sent‚ 2.577715e+06 network.recv‚578728 disk.nvme0n1.in‚ 4848.8359375" disk.dm-0.out‚38690.4638671875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 51.19140625 cpu‚24.926777201528033( disk./.usagePercent‚78.7360610553802 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚38690.4638671875% disk.nvme0n1.out‚38690.4638671875 disk.sda3.out‚ 51.19140625 disk.dm-1.out‚ 51.19140625 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 4848.8359375 disk.dm-0.in‚ 4848.8359375 disk.drbd1260.in‚0|»\™’Ô‚(:‡ ŠßèÔ÷¼£­ gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.836' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.1672 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}°­ÚdÕ‚(:Ù ŠßèÔíÌ̯ disk.nvme0n1.in‚ 4963.96875" disk.dm-0.out‚39105.3310546875 disk.sda3.out‚ 51.5703125 disk.dm-1.out‚ 51.5703125 cpu‚24.92852642342927 network.recv‚584347 disk.sda1.in‚0 disk.dm-0.in‚ 4963.96875 disk.dm-1.in‚0) disk./.usagePercent‚78.74024242643696% disk./.usageGB‚1173.2865142822266 disk.nvme0c0n1.in‚ 4963.96875% disk.nvme0n1.out‚39105.3310546875 disk.sda3.in‚0 disk.drbd1260.out‚0.234375$ memory_percent‚99.29323196411133 proc.cpu.threads‚17 network.sent‚ 2.672101e+06 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚39105.3310546875 disk.drbd1260.in‚0& proc.memory.availableMB‚ 115.796875 proc.memory.rssMB‚ 2927.15625* proc.memory.percent‚17.865943908691406 disk.sda.in‚0 disk.sda.out‚ 51.5703125 disk.sda1.out‚01v|š’Ö‚(:‡ ™ßèԱܲ gpu.0.gpu‚13.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.969' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆï Qdׂ(:Ù ™ßèÔ³€Þ· disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 52.02734375 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 4963.96875* proc.memory.percent‚17.865943908691406 network.recv‚588890) disk./.usagePercent‚78.74242745074169% disk./.usageGB‚1173.3190727233887 proc.cpu.threads‚17 disk.sda1.out‚0 network.sent‚ 2.67835e+06 disk.sda3.in‚0 disk.sda3.out‚ 52.02734375 disk.drbd1260.out‚ 0.23828125 disk.sda2.in‚0# disk.nvme0n1.out‚39328.18359375 disk.dm-0.out‚39328.18359375 disk.dm-1.out‚ 52.02734375% disk.nvme0c0n1.out‚39328.18359375 disk.nvme0n1.in‚ 4963.96875 disk.dm-0.in‚ 4963.96875$ memory_percent‚99.29325580596924( proc.memory.availableMB‚ 115.79296875 proc.memory.rssMB‚ 2927.15625 cpu‚24.92999394986632 disk.sda1.in‚0 disk.dm-1.in‚02]Æ’Ø‚(:‡ ¨ßèÔœš•® gpu.0.gpu‚45.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.004' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢•î[Ù‚(:Ð ¨ßèÔ…‚ÿ¯# disk.nvme0n1.out‚39766.99609375 disk.dm-0.in‚ 4963.96875$ memory_percent‚99.29485321044922# proc.memory.rssMB‚ 2927.16015625* proc.memory.percent‚17.865967750549316 network.sent‚ 2.684191e+06 network.recv‚593104 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.23828125 disk.nvme0c0n1.in‚ 4963.96875 disk.dm-0.out‚39766.99609375$ disk./.usageGB‚1173.444320678711 disk.dm-1.out‚52.71875% proc.memory.availableMB‚ 115.53125 cpu‚24.931479193642204 proc.cpu.threads‚17 disk.sda1.in‚0( disk./.usagePercent‚78.7508329461134 disk.sda.in‚0 disk.sda.out‚52.71875 disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0c0n1.out‚39766.99609375 disk.nvme0n1.in‚ 4963.96875 disk.sda3.out‚52.71875 disk.drbd1260.in‚0 disk.sda1.out‚0uƒ" 2Ú- _wandbruntime‚1743  _runtime‚1743ZSKÅÛ¿ $ _timestamp‚1.788489643897829e+09 _runtime‚ 1743.53830266 _step‚127 & lr-AdamW/pg1‚0.0007897335144455299 & lr-AdamW/pg2‚0.0007897335144455299 trainer/global_step‚1599ºì±RÉÜà _step‚127 & lr-AdamW/pg1‚0.0007897335144455299 & lr-AdamW/pg2‚0.0007897335144455299 trainer/global_step‚1599 $ _timestamp‚1.788489643897829e+09 _runtime‚ 1743.53830266ŽBH2Ý- _wandbruntime‚1743  _runtime‚1743EÈ‹ì'ÞæO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1637857145344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108389477425840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8876589 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206726144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚110021436299825 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24822016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1637857145344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1637084501504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35384745 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35361731 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚110020648612061 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚125245072425984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63209453 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8876061 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1637084501504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111205526537728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123535480262656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1709385437184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 _runtime‚1743.539911843 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27824706 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125244841759744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35361731 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230666240 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112842593036288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63186437 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1631958873985 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1631188225581 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15945955 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27824706 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚112843383683072 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27824708 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚112843383683072 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2‹Y©2ñ'ßëO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚112843383683072 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1637857145344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108389477425840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8876589 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206726144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚110021436299825 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24822016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1637857145344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1637084501504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35384745 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35361731 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚110020648612061 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚125245072425984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63209453 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8876061 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1637084501504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111205526537728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123535480262656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1709385437184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 _runtime‚1743.539911843 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27824706 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125244841759744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35361731 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230666240 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112842593036288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63186437 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1631958873985 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1631188225581 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15945955 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27824706 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚112843383683072 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27824708 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0€úç•`2à- _wandbruntime‚1744  _runtime‚1744AîX;áîM trainer/global_step‚1599 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 FDeviceStatsMonitor.on_train_batch_end/allocŠUû¹%ated_bytes.all.peak‚ 11892003328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125321255195648 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1638861904384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108455526822240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111273226477056 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27841582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35383442 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8881832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24836898 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110088503875671 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110087716187699 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1632960014071 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35406482 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35406482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213875712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108455543861600 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27841582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63248064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111273208474112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27841580 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35383442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15955603 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63225022 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚112912088381440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24837437 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 _runtime‚1744.507725223 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1638089247232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63248064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123610828605440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚129 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27841580 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189935616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚112912088381440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1710450530304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1638089247232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63225022 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125321469071360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584Áë#^ù&âóM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35383442 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8881832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24836898 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110088503875671 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110087716187699 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1632960014071 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35406482 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35406482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213875712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108455543861600 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27841582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63248064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111273208474112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27841580 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35383442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15955603 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63225022 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚112912088381440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24837437 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 _runtime‚1744.507725223 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1638089247232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63248064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123610828605440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚129 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27841580 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189935616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚112912088381440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1710450530304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1638089247232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63225022 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125321469071360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 trainer/global_step‚1599 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125321255195648 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1638861904384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108455526822240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111273226477056 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27841582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328Ž *2ã- _wandbruntime‚1744  _runtime‚1744µý•™WäÑ 1loss/SRqqt3/assignment_loss‚1.752563795806574 2loss/SRqqt4/assignment_loss‚1.1326678170224846 1loss/SRqqt2/detection_loss‚2.1450042979019446 _step‚130 0loss/FRt1/assignment_loss‚3.1723035199697627 1loss/SRqqt1/assignment_loss‚2.127920395183948 0loss/FBt3/assignment_loss‚0.3829404850723222 .loss/FRt3/detection_loss‚2.017517546990088 0loss/FRt2/assignment_loss‚4.7592766775804405 0loss/FRt3/assignment_loss‚2.9228786432317326 %loss/total_loss‚46.30637699557582 trainer/global_step‚1599 _runtime‚1744.509093231 2loss/FBt2/assignment_loss‚0.012046746909618378 1loss/SRqqt3/detection_loss‚1.7459605235074247 0loss/SRqqt4/detection_loss‚1.103600127372279 /loss/FBt3/detection_loss‚2.5737130492925644 1loss/SRqqt2/assignment_loss‚1.889659961549247 0loss/FBt1/assignment_loss‚0.6732780719175935 .loss/FRt2/detection_loss‚2.625908586908789 0loss/SRqqt1/detection_loss‚2.438956485640618 .loss/FBt2/detection_loss‚3.567736476659775 /loss/FBt4/detection_loss‚0.9062517240014859 0loss/FBt4/assignment_loss‚0.1550144337831322 /loss/FRt4/detection_loss‚1.8430063834502584 epoch‚1 0loss/FRt4/assignment_loss‚1.3107527607963199 .loss/FBt1/detection_loss‚2.632104776799679 .loss/FRt1/detection_loss‚2.415313708227734 % _timestamp‚1.7884896448688676e+09 +FÛ\åÖ 1loss/SRqqt3/assignment_loss‚1.752563795806574 2loss/SRqqt4/assignment_loss‚1.1326678170224846 1loss/SRqqt2/detection_loss‚2.1450042979019446 _step‚130 0loss/FRt1/assignment_loss‚3.1723035199697627 1loss/SRqqt1/assignment_loss‚2.127920395183948 0loss/FBt3/assignment_loss‚0.3829404850723222 .loss/FRt3/detection_loss‚2.017517546990088 0loss/FRt2/assignment_loss‚4.7592766775804405 0loss/FRt3/assignment_loss‚2.9228786432317326 %loss/total_loss‚46.30637699557582 trainer/global_step‚1599 _runtime‚1744.509093231 2loss/FBt2/assignment_loss‚0.012046746909618378 1loss/SRqqt3/detection_loss‚1.7459605235074247 0loss/SRqqt4/detection_loss‚1.103600127372279 /loss/FBt3/detection_loss‚2.5737130492925644 1loss/SRqqt2/assignment_loss‚1.889659961549247 0loss/FBt1/assignment_loss‚0.6732780719175935 .loss/FRt2/detection_loss‚2.625908586908789 0loss/SRqqt1/detection_loss‚2.438956485640618 .loss/FBt2/detection_loss‚3.567736476659775 /loss/FBt4/detection_loss‚0.9062517240014859 0loss/FBt4/assignment_loss‚0.1550144337831322 /loss/FRt4/detection_loss‚1.8430063834502584 epoch‚1 0loss/FRt4/assignment_loss‚1.3107527607963199 .loss/FBt1/detection_loss‚2.632104776799679 .loss/FRt1/detection_loss‚2.415313708227734 % _timestamp‚1.7884896448688676e+09‚†‚ C‘æ‚(:† ·ßèÔ‚Ñœ± gpu.0.gpu‚26.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.775' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü7´fç‚(:Û ·ßèÔ¥½à³ disk.dm-1.in‚0 disk.drbd1260.out‚ 0.2421875$ memory_percent‚99.29356575012207" proc.memory.rssMB‚ 2927.1640625 cpu‚24.932886860524665 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 4963.96875" disk.dm-0.out‚40061.8896484375 disk.sda3.out‚ 53.109375 disk.dm-1.out‚ 53.109375' proc.memory.availableMB‚ 115.7421875 proc.cpu.threads‚17 network.recv‚596702 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 4963.96875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 53.109375) disk./.usagePercent‚78.75926941839676 disk.sda1.in‚0' disk.nvme0c0n1.out‚40061.8896484375 disk.nvme0n1.in‚ 4963.96875 disk.sda.in‚0* proc.memory.percent‚17.865991592407227 network.sent‚ 2.689312e+06% disk./.usageGB‚1173.5700302124023% disk.nvme0n1.out‚40061.8896484375C[úq‘è‚(:† ÆßèÔ‚¦½¯ gpu.0.gpu‚39.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.715' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.743 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ß#ré‚(:ç ÆßèÔ˜ŠÙ² disk.dm-1.out‚ 53.5078125 disk.drbd1260.in‚0 disk.sda.out‚ 53.5078125 disk.sda2.in‚0' proc.memory.availableMB‚ 115.7421875) disk./.usagePercent‚78.75923357734197% disk./.usageGB‚1173.5694961547852 disk.dm-0.in‚ 5087.18359375" disk.dm-0.out‚40530.9482421875 disk.sda3.in‚0 disk.sda.in‚0# disk.nvme0c0n1.in‚ 5087.18359375' disk.nvme0c0n1.out‚40530.9482421875 network.recv‚602080 disk.drbd1260.out‚ 0.2421875 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.29356575012207" proc.memory.rssMB‚ 2927.1640625! disk.nvme0n1.in‚ 5087.18359375 disk.dm-1.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.865991592407227 cpu‚24.934406202983617 proc.cpu.threads‚17 network.sent‚ 2.796736e+06% disk.nvme0n1.out‚40530.9482421875 disk.sda3.out‚ 53.5078125¤ƒ#’ê‚(:‡ ÕßèÔ÷±±­ gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.281' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0562 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K rë‚(:ç ÕßèÔóÎ…²$ disk.nvme0n1.out‚40872.841796875 disk.sda3.in‚0 disk.dm-1.out‚ 54.04296875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.24609375 cpu‚24.935756900934965 network.recv‚606883) disk./.usagePercent‚78.77601128710594& disk.nvme0c0n1.out‚40872.841796875! disk.dm-0.out‚40872.841796875 disk.sda.out‚ 54.04296875 disk.sda1.in‚0* proc.memory.percent‚17.865991592407227% disk./.usageGB‚1173.8194961547852# disk.nvme0c0n1.in‚ 5087.18359375 disk.dm-1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 115.4921875 proc.cpu.threads‚17 disk.sda2.in‚0! disk.nvme0n1.in‚ 5087.18359375 disk.dm-0.in‚ 5087.18359375 disk.sda3.out‚ 54.04296875 disk.sda.in‚0$ memory_percent‚99.29509162902832" proc.memory.rssMB‚ 2927.1640625 network.sent‚ 2.80301e+06 disk.sda2.out‚0EÿÞó2ì- _wandbruntime‚1793  _runtime‚1793Ü·9æÇíÁ _runtime‚1793.665116472 _step‚131 & lr-AdamW/pg1‚0.0007890499732375841 & lr-AdamW/pg2‚0.0007890499732375841 trainer/global_step‚1649 % _timestamp‚1.7884896940246747e+09‰àïQÌîÆ & lr-AdamW/pg1‚0.0007890499732375841 & lr-AdamW/pg2‚0.0007890499732375841 trainer/global_step‚1649 % _timestamp‚1.7884896940246747e+09 _runtime‚1793.665116472 _step‚131ƒ1ˆG¹2ï- _wandbruntime‚1793  _runtime‚1793œ ÊðæO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127302873461760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16428357 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚116278632349696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65119039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25569737 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9141382 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129064647020544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚132 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113374828672933 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1682029459093 % _timestamp‚1.7884896940258908e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28668408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116277841702912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚114590523504128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114590505501184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111692782174480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1681258810689 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36473647 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65142055 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28668406 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36450633 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25570257 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16428355 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129064871395328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1761973993472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127302897401856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚114590505501184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1687336201728 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111692799213840 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 GADeviceStatsMonitor.on_train_batcheNº}è_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28668408 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1793.666438488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200434688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1687336201728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28668406 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224374784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚114590523504128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36450633 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9141900 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113374040985169 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1688108845568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚1649 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚116278632349696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1688108845568 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116277841702912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1761773558784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1793.666438488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200434688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1687336201728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28668406 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224374784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚114590523504128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36450633 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9141900 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113374040985169 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1688108845568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚1649 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚116278632349696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1688108845568 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116277841702912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1761773558784 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113374828672933 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1682029459093 % _timestamp‚1.7884896940258908e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28668408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116277841702912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚114590523504128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114590505501184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111692782174480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1681258810689 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36473647 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65142055 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28668406 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36450633 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25570257 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16428355 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129064871395328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1761973993472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784„þ“” 2ò- _wandbruntime‚1794  _runtime‚1794Þ6‘ ö&óðM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36495384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198324224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36472344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127378221804544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111758865649600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116347337048064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129141037699072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 % _timestamp‚1.7884896948999426e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16438003 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28685280 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65180666 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222264320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1689113604608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114658223443456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65180666 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129141259963392 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114658223443456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1688340947456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25584590 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127378245744640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1763014218752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113441108560807 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1683030599179 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25585124 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚114658205440512 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1682259950567 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚133 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65157624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1688340947456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1762815894528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111758848610240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1649 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116347337048064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28685282 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16438005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28685282 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36472344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113441896248779 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _runtime‚1794.541092104 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28685280 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65180666 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222264320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1689113604608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114658223443456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65180666 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129141259963392 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114658223443456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1688340947456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25584590 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127378245744640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1763014218752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113441108560807 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1683030599179 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25585124 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚114658205440512 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1682259950567 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚133 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65157624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1688340947456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1762815894528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111758848610240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1649 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116347337048064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28685282 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16438005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28685282 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36472344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113441896248779 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _runtime‚1794.541092104 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36472344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127378221804544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111758865649600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116347337048064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129141037699072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 % _timestamp‚1.7884896948999426e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16438003 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2…Rþ:2õ- _wandbruntime‚1794  _runtime‚1794gSÜ7\öÖ /loss/FBt1/detection_loss‚2.3856224976480007 % _timestamp‚1.7884896949026432e+09 1loss/FBt3/assignment_loss‚0.05426083816819301 1loss/SRqqt3/detection_loss‚1.5640021418963654 0loss/FBt4/detection_loss‚0.46991758000779965 .loss/FBt3/detection_loss‚3.072278380393982 0loss/FRt3/assignment_loss‚3.1377129706681943 2loss/SRqqt1/assignment_loss‚1.8927688287889086 1loss/SRqqt2/assignment_loss‚1.991336467365424 2loss/SRqqt3/assignment_loss‚1.7474062613387038 _runtime‚1794.542853939 _step‚134 0loss/FRt1/assignment_loss‚3.5665404349565506 /loss/FRt2/assignment_loss‚4.461877107620239 %loss/total_loss‚44.12208711434484 .loss/FRt2/detection_loss‚2.377799409000497 1loss/SRqqt2/detection_loss‚2.4939676837788687 1loss/FBt4/assignment_loss‚0.15426817090221814 .loss/FBt2/detection_loss‚2.631232738494873 epoch‚1 trainer/global_step‚1649 2loss/SRqqt4/assignment_loss‚1.0061708762576538 1loss/FBt1/assignment_loss‚0.04315140261314809 /loss/FRt3/detection_loss‚2.0995983773586797 /loss/FRt4/detection_loss‚1.9686011182873144 /loss/FRt4/assignment_loss‚0.999680300121722 1loss/FBt2/assignment_loss‚0.10230078855238389 0loss/SRqqt1/detection_loss‚2.422446457351126 1loss/SRqqt4/detection_loss‚0.7574456777689712 .loss/FRt1/detection_loss‚2.721700605005026 ¯ºa÷Û 0loss/FRt1/assignment_loss‚3.5665404349565506 /loss/FRt2/assignment_loss‚4.461877107620239 %loss/total_loss‚44.12208711434484 .loss/FRt2/detection_loss‚2.377799409000497 1loss/SRqqt2/detection_loss‚2.4939676837788687 1loss/FBt4/assignment_loss‚0.15426817090221814 .loss/FBt2/detection_loss‚2.631232738494873 epoch‚1 trainer/global_step‚1649 2loss/SRqqt4/assignment_loss‚1.0061708762576538 1loss/FBt1/assignment_loss‚0.04315140261314809 /loss/FRt3/detection_loss‚2.0995983773586797 /loss/FRt4/detection_loss‚1.9686011182873144 /loss/FRt4/assignment_loss‚0.999680300121722 1loss/FBt2/assignment_loss‚0.10230078855238389 0loss/SRqqt1/detection_loss‚2.422446457351126 1loss/SRqqt4/detection_loss‚0.7574456777689712 .loss/FRt1/detection_loss‚2.721700605005026 /loss/FBt1/detection_loss‚2.3856224976480007 % _timestamp‚1.7884896949026432e+09 1loss/FBt3/assignment_loss‚0.05426083816819301 1loss/SRqqt3/detection_loss‚1.5640021418963654 0loss/FBt4/detection_loss‚0.46991758000779965 .loss/FBt3/detection_loss‚3.072278380393982 0loss/FRt3/assignment_loss‚3.1377129706681943 2loss/SRqqt1/assignment_loss‚1.8927688287889086 1loss/SRqqt2/assignment_loss‚1.991336467365424 2loss/SRqqt3/assignment_loss‚1.7474062613387038 _runtime‚1794.542853939 _step‚134†Þ–·¢ø‚(:’ äßèÔâ·²® gpu.0.gpu‚41.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.242' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.848399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙpQnù‚(:ã äßèÔº÷ç® disk.dm-0.in‚ 5087.18359375" disk.dm-0.out‚41311.9599609375$ memory_percent‚99.29399490356445" proc.memory.rssMB‚ 2927.1640625 network.recv‚611336% disk./.usageGB‚1173.8195762634277 disk.dm-1.out‚ 54.578125 disk.drbd1260.in‚0 disk.sda.out‚ 54.578125 disk.sda2.in‚0! disk.nvme0n1.in‚ 5087.18359375 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 5087.18359375% disk.nvme0n1.out‚41311.9599609375 cpu‚24.93715472104798) disk./.usagePercent‚78.77601666326416 disk.sda3.out‚ 54.578125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚41311.9599609375& proc.memory.availableMB‚ 115.671875* proc.memory.percent‚17.865991592407227 network.sent‚ 2.808858e+06 disk.drbd1260.out‚ 0.24609375 disk.sda1.out‚0ï)»‘ú‚(:† óßèÔ›ÙÖ® gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.945 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 †Ñpû‚(:å óßèÔ°øÞ° disk.sda3.out‚ 54.9140625 disk.dm-1.out‚ 54.9140625 disk.drbd1260.out‚0.25! disk.nvme0n1.in‚ 5087.18359375 proc.cpu.threads‚17) disk./.usagePercent‚78.77601538322648% disk./.usageGB‚1173.8195571899414 disk.sda.out‚ 54.9140625 disk.sda1.out‚0& disk.nvme0n1.out‚41751.18212890625 disk.dm-0.in‚ 5087.18359375 network.sent‚ 2.918442e+06 network.recv‚616532 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚99.29440021514893( proc.memory.availableMB‚ 115.60546875* proc.memory.percent‚17.865991592407227 cpu‚24.93852298003409 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 5087.18359375( disk.nvme0c0n1.out‚41751.18212890625# disk.dm-0.out‚41751.18212890625" proc.memory.rssMB‚ 2927.1640625 disk.sda3.in‚0º$k2‘ü‚(:† ‚àèÔÑ‘–¯ gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.945 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pŽ–ný‚(:ã ‚àèÔœ£Û² disk.drbd1260.in‚0 network.sent‚ 2.921562e+06# disk.dm-0.out‚41998.17431640625 disk.drbd1260.out‚0.25 disk.sda.out‚ 55.39453125 disk.sda2.in‚0* proc.memory.percent‚17.865991592407227 disk.dm-1.out‚ 55.39453125( disk.nvme0c0n1.out‚41998.17431640625& disk.nvme0n1.out‚41998.17431640625 disk.sda1.in‚0$ memory_percent‚99.29437637329102" proc.memory.rssMB‚ 2927.1640625) disk./.usagePercent‚78.77599208654087 disk.nvme0n1.in‚ 5223.5390625 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 115.609375 cpu‚24.93986860631714 proc.cpu.threads‚17 disk.dm-1.in‚0 network.recv‚620037% disk./.usageGB‚1173.8192100524902" disk.nvme0c0n1.in‚ 5223.5390625 disk.dm-0.in‚ 5223.5390625 disk.sda3.in‚0 disk.sda3.out‚ 55.39453125<Ôd2þ- _wandbruntime‚1843  _runtime‚1843¾JÅÿ¿ & lr-AdamW/pg1‚0.0007882845462318312 & lr-AdamW/pg2‚0.0007882845462318312 trainer/global_step‚1699 # _timestamp‚1.78848974366712e+09 _runtime‚1843.307491066 _step‚135ÖkÊ€Ä trainer/global_step‚1699 # _timestamp‚1.78848974366712e+09 _runtime‚1843.307491066 _step‚135 & lr-AdamW/pg1‚0.0007882845462318312 & lr-AdamW/pg2‚0.0007882845462318312‡ôøŸ2- _wandbruntime‚1843  _runtime‚1843©Tûì'‚æO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37539535 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1737587901952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117975520470528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29512108 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119713881016320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119713881016320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚116728221046041 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 181560320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16910755 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1737587901952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117975502467584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29512108 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117975520470528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117975502467584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37562549 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26317828 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _runtime‚1843.308810825 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131070290600960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29512106 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1814477645312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114996103962480 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119713090369536 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16910757 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26318348 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚114996121001840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 % _timestamp‚1.7884897436683233e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37562549 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1738360545792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119713090369536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1738360545792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205500416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚131070314541056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1814296084992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 trainer/global_step‚1699 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9407591 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67074657 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _step‚136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132884792186368 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37539535 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9407073 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132884586685952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119713090369536 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16910757 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26318348 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚114996121001840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 % _timestamp‚1.7884897436683233e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37562549 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1738360545792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119713090369536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1738360545792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205500416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚131070314541056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1814296084992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 trainer/global_step‚1699 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9407591 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67074657 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _step‚136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132884792186368 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37539535 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9407073 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132884586685952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16910755 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1737587901952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117975502467584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29512108 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117975520470528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117975502467584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37562549 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26317828 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _runtime‚1843.308810825 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131070290600960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113ˆX's?2„- _wandbruntime‚1844  _runtime‚1844ñ 5Gg …ðM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119782585714688 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1733101184287 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1738592647680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115062187437600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1815535254016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37584286 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16920405 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132960971679232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚¦.!#28113 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26332741 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132961198137856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226458624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131145662883840 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 _step‚137 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29528982 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚118043220409856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29528980 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29528982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1815332735488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131145638943744 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67113268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37584286 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67090226 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116795288621887 % _timestamp‚1.7884897445924258e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29528980 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9412861 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118043202406912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118043202406912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2­C}û&†õM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131145638943744 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67113268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37584286 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67090226 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116795288621887 % _timestamp‚1.7884897445924258e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29528980 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9412861 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118043202406912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118043202406912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119782585714688 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1733101184287 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1738592647680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115062187437600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1815535254016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37584286 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16920405 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132960971679232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26332741 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132961198137856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226458624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131145662883840 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 _step‚137 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29528982 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚118043220409856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29528980 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29528982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1815332735488‰,Pêu2‡- _wandbruntime‚1844  _runtime‚1844$ÛâôXˆÒ 2loss/SRqqt3/assignment_loss‚1.7935582310892642 _runtime‚1844.235749823 0loss/FRt3/assignment_loss‚3.9283143913044647 .loss/FBt2/detection_loss‚2.074749324056837 /loss/FBt4/detection_loss‚0.7927781973256322 2loss/SRqqt1/assignment_loss‚1.9936483015908915 2loss/SRqqt4/assignment_loss‚0.6971135211576308 1loss/SRqqt3/detection_loss‚1.8490578948985785 0loss/FBt4/assignment_loss‚0.1388261945726408 /loss/FRt2/detection_loss‚2.4893287212952324 1loss/SRqqt4/detection_loss‚1.1060298589723452 1loss/SRqqt2/assignment_loss‚2.186395781806537 1loss/FBt2/assignment_loss‚0.19256832961562192 0loss/SRqqt1/detection_loss‚2.368348056779188 % _timestamp‚1.7884897445955517e+09 epoch‚1 0loss/FBt1/assignment_loss‚0.1958737742776672 /loss/FRt3/detection_loss‚2.6722877801341167 /loss/FRt4/detection_loss‚1.9004979758186544 .loss/FBt3/detection_loss‚4.664483904838562 %loss/total_loss‚48.85878789394303 /loss/FRt2/assignment_loss‚4.303275692722072 /loss/FRt4/assignment_loss‚1.285933087480829 .loss/FRt1/detection_loss‚2.356332164743672 1loss/SRqqt2/detection_loss‚2.5510436500821796 .loss/FBt1/detection_loss‚3.772900422414144 trainer/global_step‚1699 _step‚138 0loss/FRt1/assignment_loss‚3.4689868338730023 0loss/FBt3/assignment_loss‚0.0764558030932676órUÈ]‰× 0loss/FRt3/assignment_loss‚3.9283143913044647 .loss/FBt2/detection_loss‚2.074749324056837 /loss/FBt4/detection_loss‚0.7927781973256322 2loss/SRqqt1/assignment_loss‚1.9936483015908915 2loss/SRqqt4/assignment_loss‚0.6971135211576308 1loss/SRqqt3/detection_loss‚1.8490578948985785 0loss/FBt4/assignment_loss‚0.1388261945726408 /loss/FRt2/detection_loss‚2.4893287212952324 1loss/SRqqt4/detection_loss‚1.1060298589723452 1loss/SRqqt2/assignment_loss‚2.186395781806537 1loss/FBt2/assignment_loss‚0.19256832961562192 0loss/SRqqt1/detection_loss‚2.368348056779188 % _timestamp‚1.7884897445955517e+09 epoch‚1 0loss/FBt1/assignment_loss‚0.1958737742776672 /loss/FRt3/detection_loss‚2.6722877801341167 /loss/FRt4/detection_loss‚1.9004979758186544 .loss/FBt3/detection_loss‚4.664483904838562 %loss/total_loss‚48.85878789394303 /loss/FRt2/assignment_loss‚4.303275692722072 /loss/FRt4/assignment_loss‚1.285933087480829 .loss/FRt1/detection_loss‚2.356332164743672 1loss/SRqqt2/detection_loss‚2.5510436500821796 .loss/FBt1/detection_loss‚3.772900422414144 trainer/global_step‚1699 _step‚138 0loss/FRt1/assignment_loss‚3.4689868338730023 0loss/FBt3/assignment_loss‚0.0764558030932676 2loss/SRqqt3/assignment_loss‚1.7935582310892642 _runtime‚1844.235749823Š¡Ç½’Š‚(:‡ ‘àèÔ•Šé® gpu.0.gpu‚44.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.623' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 5x=p‹‚(:å ‘àèÔ‡’´´ disk.dm-1.out‚ 55.7734375 network.sent‚ 2.940793e+06$ disk./.usageGB‚1173.796230316162 disk.nvme0n1.in‚ 5223.5390625& disk.nvme0n1.out‚42148.21044921875 disk.dm-0.in‚ 5223.5390625# disk.dm-0.out‚42148.21044921875 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 5223.5390625( disk.nvme0c0n1.out‚42148.21044921875 disk.sda3.out‚ 55.7734375 disk.drbd1260.in‚0 disk.sda.out‚ 55.7734375 disk.sda1.out‚0$ memory_percent‚99.29590225219727& proc.memory.availableMB‚ 115.359375 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.25390625 disk.sda.in‚0 disk.sda1.in‚0 cpu‚24.94120121871185 proc.cpu.threads‚17 network.recv‚624737) disk./.usagePercent‚78.77444989715461 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2927.1640625* proc.memory.percent‚17.865991592407227i\Ö’Œ‚(:‡  àèÔœœ»± gpu.0.gpu‚21.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.748' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°î&n‚(:ã  àèԛܹ· proc.cpu.threads‚17$ disk./.usageGB‚1173.796230316162 disk.sda3.in‚0 disk.sda.out‚ 56.328125 disk.sda2.in‚0 disk.dm-0.in‚ 5223.5390625# disk.dm-0.out‚42737.40966796875$ memory_percent‚99.29461479187012 network.sent‚ 3.036057e+06 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 5223.5390625& disk.nvme0n1.out‚42737.40966796875' proc.memory.availableMB‚ 115.5703125) disk./.usagePercent‚78.77444989715461 disk.dm-1.out‚ 56.328125 disk.drbd1260.out‚ 0.25390625 disk.sda2.out‚0 disk.nvme0n1.in‚ 5223.5390625" proc.memory.rssMB‚ 2927.1640625* proc.memory.percent‚17.865991592407227 cpu‚24.94247680792793 network.recv‚630802 disk.sda3.out‚ 56.328125 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚42737.40966796875=æ`Ž‚(:’ ¯àèÔ¿™ø¯ gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚126.786' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.357200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+{µn‚(:ã ¯àèÔñÂÌ° network.sent‚ 3.041421e+06) disk./.usagePercent‚78.77602357546758$ disk./.usageGB‚1173.819679260254 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 5223.5390625! disk.dm-0.out‚43033.298828125 disk.sda3.out‚ 56.62890625 disk.drbd1260.out‚ 0.2578125 disk.sda1.in‚0& disk.nvme0c0n1.out‚43033.298828125 disk.nvme0n1.in‚ 5223.5390625$ disk.nvme0n1.out‚43033.298828125 network.recv‚634536 disk.dm-1.in‚0 disk.dm-1.out‚ 56.62890625 disk.dm-0.in‚ 5223.5390625' proc.memory.availableMB‚ 115.8203125" proc.memory.rssMB‚ 2927.1640625 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 56.62890625$ memory_percent‚99.29308891296387* proc.memory.percent‚17.865991592407227 cpu‚24.943997319882925gÖ\‚(:„ ¾àèÔèÅž³ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.46' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑrE„r‘‚(:ç ¾àèÔÛ˃½% disk./.usageGB‚1173.8196983337402 disk.dm-1.out‚ 56.94921875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0$ disk.nvme0n1.out‚43398.994140625 disk.dm-0.in‚ 5336.58984375* proc.memory.percent‚17.865991592407227 network.sent‚ 3.046269e+06 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2578125 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚43398.994140625! disk.nvme0n1.in‚ 5336.58984375 network.recv‚637622) disk./.usagePercent‚78.77602485550526 disk.sda3.in‚0 disk.sda.out‚ 56.94921875# disk.nvme0c0n1.in‚ 5336.58984375! disk.dm-0.out‚43398.994140625" proc.memory.rssMB‚ 2927.1640625 cpu‚24.94524009014972 disk.sda3.out‚ 56.94921875$ memory_percent‚99.29320812225342( proc.memory.availableMB‚ 115.80078125 proc.cpu.threads‚17Óå/2’- _wandbruntime‚1894  _runtime‚1894Y«ƒ‰Æ“À trainer/global_step‚1749 % _timestamp‚1.7884897950021622e+09 _runtime‚ 1894.64252074 _step‚139 & lr-AdamW/pg1‚0.0007874373929421124 & lr-AdamW/pg2‚0.0007874373929421124yÁ/˔Štrainer/global_step‚1749 % _timestamp‚1.7884897950021622e+09 _runtime‚ 1894.64252074 _step‚139 & lr-AdamW/pg1‚0.0007874373929421124 & lr-AdamW/pg2‚0.0007874373929421124‹($ë52•- _wandbruntime‚1894  _runtime‚1894¦(Ï–åO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1782170629309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226471936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123149129682944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136704872576000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 $ _timestamp‚1.788489795003275e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30355806 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38651451 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38628437 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121360499433984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30355808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118299425750480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38651451 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1787839602176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9673390 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202531840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123148339036160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136704646104064 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118299442789840 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123149129682944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1866938363904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38628437 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123148339036160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1867140895744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121360517436928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27066547 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.%ú¡Ý small_pool.freed‚ 1781399980905 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 _runtime‚1894.643835549 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68984243 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _step‚140 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121360499433984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69007259 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1788612246016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121360517436928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30355808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134837731680256 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚1749‡%ýGð'—êO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30355806 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1787839602176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9673390 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202531840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123148339036160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136704646104064 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118299442789840 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123149129682944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1866938363904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38628437 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123148339036160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1867140895744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121360517436928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27066547 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1781399980905 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 _runtime‚1894.643835549 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68984243 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _step‚140 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121360499433984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69007259 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1788612246016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121360517436928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30355808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134837731680256 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚1749 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1782170629309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226471936 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123149129682944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136704872576000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 $ _timestamp‚1.788489795003275e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30355806 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38651451 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38628437 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121360499433984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30355808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118299425750480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38651451 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚121428199373312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚121428217376256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17402805 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136781258720256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9678208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1867978275840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚118365509225600 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚123217834381312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1783171769395 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120147893307023 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69022828 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38650148 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69045870 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1788844347904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30372682 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30372680 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69022828 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17402803 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1788844347904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121428217376256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134913056082944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _runtime‚1895.576963807 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27081011 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224361472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9678745 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69045870 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134913080023040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38673188 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚118365492186240 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1868178697216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38673188 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136781034358784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123217834381312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005\%!é%šôM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1789617005056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚120148680994995 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚121428199373312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚121428217376256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17402805 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136781258720256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9678208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1867978275840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚118365509225600 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚123217834381312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1783171769395 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120147893307023 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69022828 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38650148 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69045870 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1788844347904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30372682 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30372680 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69022828 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17402803 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1788844347904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121428217376256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134913056082944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _runtime‚1895.576963807 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27081011 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224361472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9678745 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69045870 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134913080023040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38673188 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚118365492186240 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1868178697216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38673188 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136781034358784 WCDeviceStatsMonitor.on_train_batch_end/allocasÊÛ}ted_bytes.all.allocated‚123217834381312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005¬^2›- _wandbruntime‚1895  _runtime‚1895+DÁÊWœÑ 1loss/SRqqt1/assignment_loss‚2.030336376956918 1loss/FBt3/assignment_loss‚0.24468190429939163 _runtime‚1895.578240113 epoch‚1 trainer/global_step‚1749 /loss/FRt3/assignment_loss‚3.488032521203507 2loss/SRqqt3/assignment_loss‚1.9193207019141743 1loss/FBt1/assignment_loss‚0.13579539409838617 -loss/FBt3/detection_loss‚2.43341330687205 1loss/FBt2/assignment_loss‚0.24736311221178653 _step‚142 0loss/FRt1/assignment_loss‚3.4548470865596426 0loss/FRt4/assignment_loss‚1.6105994788967832 2loss/SRqqt2/assignment_loss‚2.0600535138636022 /loss/FRt4/detection_loss‚1.6754347265374905 1loss/SRqqt2/detection_loss‚2.3211890152982764 /loss/FBt4/detection_loss‚0.7543706301114782 -loss/FRt1/detection_loss‚2.50368050689047 &loss/total_loss‚45.417148869235824 % _timestamp‚1.7884897959376686e+09 1loss/FBt4/assignment_loss‚0.21421460306122642 /loss/FRt2/assignment_loss‚4.077605718060544 2loss/SRqqt4/assignment_loss‚0.9809836625307798 0loss/SRqqt1/detection_loss‚2.209713750264861 .loss/FRt3/detection_loss‚2.633424437323282 /loss/FBt1/detection_loss‚2.0286229312419892 /loss/FBt2/detection_loss‚3.0235500782728195 .loss/FRt2/detection_loss‚2.464431296837957 1loss/SRqqt3/detection_loss‚1.7006763473153115 .loss/SRqqt4/detection_loss‚1.2048077686131›tœƒ\Ö 1loss/SRqqt1/assignment_loss‚2.030336376956918 1loss/FBt3/assignment_loss‚0.24468190429939163 _runtime‚1895.578240113 epoch‚1 trainer/global_step‚1749 /loss/FRt3/assignment_loss‚3.488032521203507 2loss/SRqqt3/assignment_loss‚1.9193207019141743 1loss/FBt1/assignment_loss‚0.13579539409838617 -loss/FBt3/detection_loss‚2.43341330687205 1loss/FBt2/assignment_loss‚0.24736311221178653 _step‚142 0loss/FRt1/assignment_loss‚3.4548470865596426 0loss/FRt4/assignment_loss‚1.6105994788967832 2loss/SRqqt2/assignment_loss‚2.0600535138636022 /loss/FRt4/detection_loss‚1.6754347265374905 1loss/SRqqt2/detection_loss‚2.3211890152982764 /loss/FBt4/detection_loss‚0.7543706301114782 -loss/FRt1/detection_loss‚2.50368050689047 &loss/total_loss‚45.417148869235824 % _timestamp‚1.7884897959376686e+09 1loss/FBt4/assignment_loss‚0.21421460306122642 /loss/FRt2/assignment_loss‚4.077605718060544 2loss/SRqqt4/assignment_loss‚0.9809836625307798 0loss/SRqqt1/detection_loss‚2.209713750264861 .loss/FRt3/detection_loss‚2.633424437323282 /loss/FBt1/detection_loss‚2.0286229312419892 /loss/FBt2/detection_loss‚3.0235500782728195 .loss/FRt2/detection_loss‚2.464431296837957 1loss/SRqqt3/detection_loss‚1.7006763473153115 .loss/SRqqt4/detection_loss‚1.2048077686131Žmܽõ’ž‚(:‡ ÍàèÔôúì­ gpu.0.gpu‚31.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.517' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5034 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®" &_Ÿ‚(:Ô ÍàèÔú÷® disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0# disk.nvme0n1.out‚43669.86328125 network.recv‚642080 disk.sda3.in‚0 disk.sda3.out‚57.375 disk.drbd1260.in‚0 disk.sda.out‚57.375 disk.dm-0.in‚ 5336.58984375$ memory_percent‚99.29509162902832' proc.memory.availableMB‚ 115.4921875 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 5336.58984375% disk.nvme0c0n1.out‚43669.86328125! disk.nvme0n1.in‚ 5336.58984375* proc.memory.percent‚17.866015434265137 cpu‚24.94639437811728) disk./.usagePercent‚78.77602357546758 disk.drbd1260.out‚ 0.26171875 disk.sda.in‚0 disk.dm-0.out‚43669.86328125# proc.memory.rssMB‚ 2927.16796875 proc.cpu.threads‚17 network.sent‚ 3.05172e+06$ disk./.usageGB‚1173.819679260254 disk.dm-1.in‚0 disk.dm-1.out‚57.375„%'’ ‚(:‡ ÜàèÔ¿ëÙ® gpu.0.gpu‚37.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.746' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.1492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈKÈo¡‚(:ä ÜàèÔÂḳ disk.sda.in‚0 disk.sda1.in‚0 network.sent‚ 3.160328e+06 network.recv‚648151 disk.dm-1.out‚ 57.86328125 disk.drbd1260.out‚ 0.26171875 disk.dm-0.out‚ 43940.5078125$ memory_percent‚99.29511547088623 disk.sda3.in‚0 disk.sda3.out‚ 57.86328125 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 5336.58984375( proc.memory.availableMB‚ 115.48828125# proc.memory.rssMB‚ 2927.16796875* proc.memory.percent‚17.866015434265137 cpu‚24.947679758818435 disk.sda.out‚ 57.86328125 disk.sda1.out‚0) disk./.usagePercent‚78.77458353308748% disk./.usageGB‚1173.7982215881348 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 5336.58984375 disk.sda2.in‚0$ disk.nvme0c0n1.out‚ 43940.5078125! disk.nvme0n1.in‚ 5336.58984375" disk.nvme0n1.out‚ 43940.5078125 proc.cpu.threads‚17l^¬ÿ¢‚(:’ ëàèÔ¬•Í® gpu.0.gpu‚28.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.591' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.918200000000002 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.—8w£‚(:ì ëàèԌٖ³ disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0# disk.nvme0c0n1.in‚ 5336.58984375 disk.sda3.out‚ 58.24609375 proc.cpu.threads‚17 network.recv‚651916) disk./.usagePercent‚78.77626140646689 disk.dm-1.out‚ 58.24609375( disk.nvme0c0n1.out‚44360.53173828125& disk.nvme0n1.out‚44360.53173828125 disk.dm-0.in‚ 5336.58984375' proc.memory.availableMB‚ 116.1640625 network.sent‚ 3.165394e+06% disk./.usageGB‚1173.8232231140137 disk.drbd1260.out‚0.265625 disk.sda.out‚ 58.24609375 disk.sda1.in‚0! disk.nvme0n1.in‚ 5336.58984375# disk.dm-0.out‚44360.53173828125 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚99.29099082946777# proc.memory.rssMB‚ 2927.16796875* proc.memory.percent‚17.866015434265137 cpu‚24.94894495292595Äœ÷Ñ2¤- _wandbruntime‚1945  _runtime‚1945^bg‚Ä¥¾ _runtime‚1945.471747008 _step‚143 % lr-AdamW/pg1‚0.000786508689913903 % lr-AdamW/pg2‚0.000786508689913903 trainer/global_step‚1799 $ _timestamp‚1.788489845831349e+09 õfɦà _runtime‚1945.471747008 _step‚143 % lr-AdamW/pg1‚0.000786508689913903 % lr-AdamW/pg2‚0.000786508689913903 trainer/global_step‚1799 $ _timestamp‚1.788489845831349e+09°ën›2§- _wandbruntime‚1945  _runtime‚1945º{ ì'¨æO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123435005792257 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126583587702784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9939126 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚124745496400384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31199506 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚124745514403328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 _step‚144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1919484006400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1838863946240 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138605148819456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226471936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39717339 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31199506 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39740353 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39717339 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31199506 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39740353 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126583587702784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9939126 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚124745496400384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31199506 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚124745514403328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 _step‚144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1919484006400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1838863946240 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138605148819456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226471936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚126653083047936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140601026352128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1920332962816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31216382 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196227072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123501285680131 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStH-›oatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1832471705891 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9944915 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70955430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚124813196339712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27829565 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17885203 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1920529189888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39739050 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123502073368103 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚145 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚126653083047936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70955430 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31216382 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1839096048128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚126652292387840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚124813196339712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70978472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39739050 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138680497162240 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27830120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1839868705280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1839868705280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚124813214342656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17885205 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9944362 % _timestamp‚1.7884898467109082e+09 _runtime‚ 1946.35190436 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220167168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960¢‘õ“ú&¬ôM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31216382 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1839096048128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚126652292387840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚124813196339712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70978472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39739050 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138680497162240 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27830120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1839868705280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1839868705280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚124813214342656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17885205 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9944362 % _timestamp‚1.7884898467109082e+09 _runtime‚ 1946.35190436 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220167168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31216380 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70978472 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140600806184960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 trainer/global_step‚1799 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121668813974240 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚126653083047936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140601026352128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1920332962816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31216382 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196227072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123501285680131 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1832471705891 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9944915 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70955430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚124813196339712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27829565 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17885203 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1920529189888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39739050 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123502073368103 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚145 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚126653083047936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70955430 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0‘¹ÂèÉ2­- _wandbruntime‚1946  _runtime‚1946¯¨«ß\®Ö 0loss/FRt4/assignment_loss‚1.1082098855162568 1loss/SRqqt1/assignment_loss‚2.265083545446396 1loss/FBt2/assignment_loss‚0.21124767349098963 /loss/FRt1/detection_loss‚2.2855459208841675 1loss/SRqqt1/detection_loss‚2.3938113510608674 /loss/FBt2/detection_loss‚2.9106548854282925 trainer/global_step‚1799 2loss/SRqqt4/assignment_loss‚1.0676683486047862 0loss/FBt3/assignment_loss‚0.2653778266782562 1loss/FBt4/assignment_loss‚0.19385964144728743 1loss/SRqqt2/detection_loss‚2.5309751947720844 % _timestamp‚1.7884898467136228e+09 _step‚146 0loss/FRt2/assignment_loss‚3.6512299299240114 1loss/FBt1/assignment_loss‚0.17629957835145302 .loss/FRt4/detection_loss‚1.326794179472721 epoch‚1 /loss/FRt1/assignment_loss‚3.662721006958573 0loss/FRt3/assignment_loss‚3.1406977447596462 2loss/SRqqt2/assignment_loss‚2.2473578388492266 .loss/FRt2/detection_loss‚2.706702905148268 /loss/FRt3/detection_loss‚2.2762434767051176 1loss/SRqqt4/detection_loss‚0.9905135706337034 %loss/total_loss‚46.09585414687258 1loss/SRqqt3/detection_loss‚1.5225697518843744 _runtime‚1946.353831321 2loss/SRqqt3/assignment_loss‚1.7964075807895925 .loss/FBt1/detection_loss‚3.092562198638916 /loss/FBt3/detection_loss‚3.3325725893179574 .loss/FBt4/detection_loss‚0.940747522109632T”la¯Û 1loss/SRqqt3/detection_loss‚1.5225697518843744 _runtime‚1946.353831321 2loss/SRqqt3/assignment_loss‚1.7964075807895925 .loss/FBt1/detection_loss‚3.092562198638916 /loss/FBt3/detection_loss‚3.3325725893179574 .loss/FBt4/detection_loss‚0.940747522109632 0loss/FRt4/assignment_loss‚1.1082098855162568 1loss/SRqqt1/assignment_loss‚2.265083545446396 1loss/FBt2/assignment_loss‚0.21124767349098963 /loss/FRt1/detection_loss‚2.2855459208841675 1loss/SRqqt1/detection_loss‚2.3938113510608674 /loss/FBt2/detection_loss‚2.9106548854282925 trainer/global_step‚1799 2loss/SRqqt4/assignment_loss‚1.0676683486047862 0loss/FBt3/assignment_loss‚0.2653778266782562 1loss/FBt4/assignment_loss‚0.19385964144728743 1loss/SRqqt2/detection_loss‚2.5309751947720844 % _timestamp‚1.7884898467136228e+09 _step‚146 0loss/FRt2/assignment_loss‚3.6512299299240114 1loss/FBt1/assignment_loss‚0.17629957835145302 .loss/FRt4/detection_loss‚1.326794179472721 epoch‚1 /loss/FRt1/assignment_loss‚3.662721006958573 0loss/FRt3/assignment_loss‚3.1406977447596462 2loss/SRqqt2/assignment_loss‚2.2473578388492266 .loss/FRt2/detection_loss‚2.706702905148268 /loss/FRt3/detection_loss‚2.2762434767051176 1loss/SRqqt4/detection_loss‚0.9905135706337034 %loss/total_loss‚46.09585414687258’«ïw°‚(:’ úàèÔ•‚„° gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.129599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03ÄO p±‚(:å úàèÔž™â± disk.dm-1.out‚ 58.66015625 disk.drbd1260.out‚0.265625# proc.memory.rssMB‚ 2927.16796875 network.recv‚655439& disk.nvme0n1.out‚44775.62158203125 disk.dm-0.in‚ 5501.203125# disk.dm-0.out‚44775.62158203125 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚99.29273128509521) disk./.usagePercent‚78.77468081595049 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 5501.203125 disk.drbd1260.in‚0 disk.sda1.in‚0* proc.memory.percent‚17.866015434265137 cpu‚24.950218797223 proc.cpu.threads‚17 disk.sda2.in‚0( disk.nvme0c0n1.out‚44775.62158203125 disk.sda3.out‚ 58.66015625 disk.dm-1.in‚0 disk.sda.out‚ 58.66015625( proc.memory.availableMB‚ 115.87890625 network.sent‚ 3.170241e+06% disk./.usageGB‚1173.7996711730957 disk.nvme0n1.in‚ 5501.203125¸.Ä‘²‚(:† ‰áèÔÒ„§® gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.479' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8958 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p•˜kl³‚(:á ‰áèÔÜúß® disk.sda.in‚0 disk.sda1.out‚0 cpu‚24.951394157369354 network.recv‚662137 disk.dm-0.in‚ 5501.203125 disk.dm-1.out‚ 59.09765625 disk.drbd1260.out‚ 0.2734375 disk.sda.out‚ 59.09765625 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚45407.0712890625" disk.dm-0.out‚45407.0712890625 disk.sda3.out‚ 59.09765625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 5501.203125# proc.memory.rssMB‚ 2927.16796875 proc.cpu.threads‚17 network.sent‚ 3.279012e+06% disk./.usageGB‚1173.8231201171875 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 5501.203125% disk.nvme0n1.out‚45407.0712890625$ memory_percent‚99.29122924804688# proc.memory.availableMB‚116.125* proc.memory.percent‚17.866015434265137) disk./.usagePercent‚78.77625449426347é§}p’´‚(:‡ ˜áèÔ´©¤­ gpu.0.gpu‚23.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.969' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`&ó4pµ‚(:å ˜áèÔªåÌ® disk.drbd1260.out‚ 0.2734375 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚24.952479830917618 network.recv‚666714% disk./.usageGB‚1173.8232536315918! disk.nvme0c0n1.in‚ 5501.203125" disk.dm-0.out‚45653.6220703125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 5501.203125 disk.sda3.out‚ 59.57421875 disk.drbd1260.in‚0 disk.sda.out‚ 59.57421875 disk.sda2.out‚0$ memory_percent‚99.29280281066895# proc.memory.rssMB‚ 2927.16796875* proc.memory.percent‚17.866015434265137 network.sent‚ 3.285069e+06% disk.nvme0n1.out‚45653.6220703125 disk.dm-0.in‚ 5501.203125 disk.sda2.in‚0' proc.memory.availableMB‚ 115.8671875 proc.cpu.threads‚17) disk./.usagePercent‚78.77626345452717' disk.nvme0c0n1.out‚45653.6220703125 disk.dm-1.out‚ 59.57421875xdîž’¶‚(:‡ §áèԵ﩮 gpu.0.gpu‚29.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.409' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á¾yd·‚(:Ù §áèÔ…ÍÔ° disk.sda1.in‚0 network.sent‚ 3.290043e+06% disk./.usageGB‚1173.0719375610352 disk.sda3.in‚0 disk.sda.out‚59.84375' disk.nvme0c0n1.out‚45894.6005859375 disk.dm-0.in‚ 5501.203125# proc.memory.rssMB‚ 2927.16796875) disk./.usagePercent‚78.72584200263594 disk.sda3.out‚59.84375 disk.dm-1.out‚59.84375 disk.sda2.in‚0 disk.nvme0n1.in‚ 5501.203125* proc.memory.percent‚17.866015434265137 cpu‚24.953614743825177 proc.cpu.threads‚17 network.recv‚670462 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.27734375 disk.sda.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚45894.6005859375 disk.drbd1260.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 5501.203125" disk.dm-0.out‚45894.6005859375$ memory_percent‚99.29275512695312# proc.memory.availableMB‚115.8756k#¥2¸- _wandbruntime‚1995  _runtime‚1995¬Y\ŒÄ¹¾ & lr-AdamW/pg2‚0.0007854986306875224 trainer/global_step‚1849 % _timestamp‚1.7884898955844681e+09 _runtime‚ 1995.224866 _step‚147 & lr-AdamW/pg1‚0.0007854986306875224²&á#ɺà & lr-AdamW/pg2‚0.0007854986306875224 trainer/global_step‚1849 % _timestamp‚1.7884898955844681e+09 _runtime‚ 1995.224866 _step‚147 & lr-AdamW/pg1‚0.0007854986306875224“Bºï2»- _wandbruntime‚1995  _runtime‚1995ïûúl7#¼çO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚128130493366784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144344312239104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32043208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130018836369408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 % _timestamp‚1.7884898955857236e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚130019627016192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚126788398165365 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18357957 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32043206 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18357955 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130018836369408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40829255 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚124906086365840 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚128130511369728 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.pˆgζeak‚39 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚128130511369728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚128130493366784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1889115646464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10205740 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ÑCvLò'½ìO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚126788398165365 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18357957 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32043206 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18357955 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130018836369408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40829255 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚124906086365840 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚128130511369728 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚128130511369728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚128130493366784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1889115646464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10205740 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚128130493366784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144344312239104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32043208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130018836369408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 % _timestamp‚1.7884898955857236e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚130019627016192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046”®™±2¾- _wandbruntime‚1996  _runtime‚1996¼«yÃ÷&¿ñM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10211548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1890120405504 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28578623 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1889347748352 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40827952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚149 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72888032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144420706923520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32060082 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚128198193306112 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130087541054464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142447914301440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚128198193306112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222264320 _runtime‚1996.245631158 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72911074 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18367603 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72888032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32060080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚130088331714560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1890120405504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788489896604316e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1889347748352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚128198211309056 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10211020 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚124972152801600 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1972816562176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚126855465741211 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1883312939611 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72911074 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40850992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40850992 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28579153 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198324224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚128198211309056 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32060082 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32060080A ´lü&ÀöM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130087541054464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142447914301440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚128198193306112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222264320 _runtime‚1996.245631158 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72911074 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18367603 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72888032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32060080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚130088331714560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1890120405504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788489896604316e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1889347748352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚128198211309056 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10211020 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚124972152801600 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1972816562176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚126855465741211 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1883312939611 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72911074 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40850992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40850992 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28579153 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198324224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚128198211309056 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32060082 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32060080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10211548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1890120405504 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28578623 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1889347748352 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40827952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚149 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72888032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144420706923520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32060082 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚128198193306112 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516•º,áÀ2Á- _wandbruntime‚1996  _runtime‚1996‹ÐñžÐÂÛ 1loss/FBt2/assignment_loss‚0.09068009971330564 1loss/FBt4/assignment_loss‚0.06752346103639457 2loss/SRqqt3/assignment_loss‚1.5939086120289105 &loss/total_loss‚44.366992591051556 2loss/SRqqt4/assignment_loss‚0.9815017650253139 /loss/FBt4/detection_loss‚0.5985195558369923 2loss/SRqqt2/assignment_loss‚1.9566827362233943 /loss/FRt3/detection_loss‚2.3473263571456986 1loss/SRqqt4/detection_loss‚1.0319230947643518 /loss/FBt1/detection_loss‚2.4887607097625732 % _timestamp‚1.7884898966071105e+09 _step‚150 1loss/FBt1/assignment_loss‚0.13712866038349603 /loss/FRt1/detection_loss‚2.6281651092130085 epoch‚1 trainer/global_step‚1849 _runtime‚1996.247797722 2loss/SRqqt1/assignment_loss‚2.1828820640664723 /loss/FRt2/detection_loss‚2.3897246049671637 1loss/SRqqt1/detection_loss‚2.1132941934324445 1loss/SRqqt2/detection_loss‚2.0736466043374757 1loss/SRqqt3/detection_loss‚1.6682398305489468 /loss/FBt3/detection_loss‚2.4031705498695373 /loss/FRt2/assignment_loss‚3.724338008136284 0loss/FRt4/assignment_loss‚1.4893315243721008 /loss/FRt4/detection_loss‚1.6144612231850624 /loss/FRt1/assignment_loss‚4.213614953118701 0ïT„‘loss/FRt3/assignment_loss‚3.0060990829857026 1loss/FBt3/assignment_loss‚0.14226387856251677 .loss/FBt2/detection_loss‚3.423805912335714¶«uâfÃà 1loss/FBt2/assignment_loss‚0.09068009971330564 1loss/FBt4/assignment_loss‚0.06752346103639457 2loss/SRqqt3/assignment_loss‚1.5939086120289105 &loss/total_loss‚44.366992591051556 2loss/SRqqt4/assignment_loss‚0.9815017650253139 /loss/FBt4/detection_loss‚0.5985195558369923 2loss/SRqqt2/assignment_loss‚1.9566827362233943 /loss/FRt3/detection_loss‚2.3473263571456986 1loss/SRqqt4/detection_loss‚1.0319230947643518 /loss/FBt1/detection_loss‚2.4887607097625732 % _timestamp‚1.7884898966071105e+09 _step‚150 1loss/FBt1/assignment_loss‚0.13712866038349603 /loss/FRt1/detection_loss‚2.6281651092130085 epoch‚1 trainer/global_step‚1849 _runtime‚1996.247797722 2loss/SRqqt1/assignment_loss‚2.1828820640664723 /loss/FRt2/detection_loss‚2.3897246049671637 1loss/SRqqt1/detection_loss‚2.1132941934324445 1loss/SRqqt2/detection_loss‚2.0736466043374757 1loss/SRqqt3/detection_loss‚1.6682398305489468 /loss/FBt3/detection_loss‚2.4031705498695373 /loss/FRt2/assignment_loss‚3.724338008136284 0loss/FRt4/assignment_loss‚1.4893315243721008 /loss/FRt4/detection_loss‚1.6144612231850624 /loss/FRt1/assignment_loss‚4.213614953118701 0loss/FRt3/assignment_loss‚3.0060990829857026 1loss/FBt3/assignment_loss‚0.14226387856251677 .loss/FBt2/detection_loss‚3.423805912335714–²ÄÄ‚(:’ ¶áèÔ쬮® gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚131.209' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.241799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸‹¾fmÅ‚(:â ¶áèÔù𨯠disk.drbd1260.out‚ 0.27734375 disk.sda.out‚60.46875 disk.sda1.in‚0# memory_percent‚99.2954969406128% disk./.usageGB‚1173.0717391967773 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 5693.65234375% disk.nvme0n1.out‚46399.6396484375( proc.memory.availableMB‚ 115.42578125* proc.memory.percent‚17.866015434265137 cpu‚24.95457919772107 proc.cpu.threads‚17) disk./.usagePercent‚78.72582869024416 disk.drbd1260.in‚0" disk.dm-0.out‚46399.6396484375# proc.memory.rssMB‚ 2927.16796875 disk.sda3.out‚60.46875 disk.dm-1.out‚60.46875 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚46399.6396484375! disk.nvme0n1.in‚ 5693.65234375 disk.dm-0.in‚ 5693.65234375 disk.sda1.out‚0 network.sent‚ 3.295062e+06 network.recv‚673832 disk.dm-1.in‚0>ï¾™‘Æ‚(:† ÅáèÔ÷—®­ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚165.025' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„‘!«qÇ‚(:æ ÅáèÔò‰í¯ disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 60.83984375 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚46902.427734375 network.recv‚680441 disk.sda3.out‚ 60.83984375$ disk.nvme0n1.out‚46902.427734375! disk.dm-0.out‚46786.052734375# proc.memory.rssMB‚ 2927.16796875* proc.memory.percent‚17.866015434265137 disk.sda3.in‚0 disk.dm-0.in‚ 5702.61328125$ memory_percent‚99.29344654083252( proc.memory.availableMB‚ 115.76171875 cpu‚24.95548341725595 proc.cpu.threads‚17) disk./.usagePercent‚78.65285732671444% disk./.usageGB‚1171.9844131469727 disk.dm-1.out‚ 60.83984375 disk.drbd1260.out‚0.28125 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 5702.61328125! disk.nvme0n1.in‚ 5702.61328125 network.sent‚ 3.403824e+06ž½ñÎ’È‚(:‡ ÔáèÔĵá® gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.344' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2688 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 )-½bÉ‚(:× ÔáèÔÓÍ…² network.recv‚684797 disk.sda.out‚61.375 disk.sda2.in‚0& disk.nvme0c0n1.out‚47235.779296875 disk.sda3.in‚0 disk.dm-1.out‚61.375 disk.drbd1260.out‚0.28125( proc.memory.availableMB‚ 115.51953125) disk./.usagePercent‚78.65443586917057 disk.sda1.in‚0 disk.sda1.out‚0! disk.dm-0.out‚47235.779296875 disk.sda3.out‚61.375* proc.memory.percent‚17.866015434265137 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 5702.61328125! disk.nvme0n1.in‚ 5702.61328125$ disk.nvme0n1.out‚47235.779296875 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚99.29492473602295 network.sent‚ 3.409569e+06% disk./.usageGB‚1172.0079345703125 disk.dm-0.in‚ 5702.61328125# proc.memory.rssMB‚ 2927.16796875 cpu‚24.95633505601501b*Œ2Ê- _wandbruntime‚2041  _runtime‚2041\àííÇËÁ & lr-AdamW/pg1‚0.0007844074257577994 & lr-AdamW/pg2‚0.0007844074257577994 trainer/global_step‚1899 % _timestamp‚1.7884899421181846e+09 _runtime‚2041.758579853 _step‚151”S@lÌÌÆ & lr-AdamW/pg1‚0.0007844074257577994 & lr-AdamW/pg2‚0.0007844074257577994 trainer/global_step‚1899 % _timestamp‚1.7884899421181846e+09 _runtime‚2041.758579853 _step‚151—™Û@–2Í- _wandbruntime‚2041  _runtime‚2041ð‡×Èî'ÎèO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131515490333184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32886908 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213889024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1931611736229 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1932382384633 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133454875682816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133454085036032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10472245 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1939367346688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚130141002850709 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2024474047488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131515508336128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32886906 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41895143 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189948928 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1939367346688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131515490333184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148164457145344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128209408153840 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41918157 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133454085036032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74782049 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133454875682816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1938594702848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148164243256320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41895143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _runtime‚2041.759777501 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130141790538473 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32886908 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10472784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41918157 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18840357 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131515508336128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1938594702848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146139959157760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1899ø¯à’ó'ÏíO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1899 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131515490333184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32886908 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213889024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1931611736229 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1932382384633 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133454875682816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133454085036032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10472245 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1939367346688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚130141002850709 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2024474047488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131515508336128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32886906 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41895143 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189948928 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1939367346688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131515490333184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148164457145344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128209408153840 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41918157 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133454085036032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74782049 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133454875682816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1938594702848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148164243256320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41895143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _runtime‚2041.759777501 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130141790538473 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32886908 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10472784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41918157 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18840357 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131515508336128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1938594702848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146139959157760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840˜ïF2Ð- _wandbruntime‚2042  _runtime‚2042¹i–ßÑòM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133522789721088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74843676 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41939894 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1939599448576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1940372105728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74843676 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220167168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146215307500544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74820634 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32903782 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚146215331440640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884899429690204e+09 _step‚153 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32903782 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131583208275456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1939599448576 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74820634 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148240642614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18850005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1932612876107 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2025335113728 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18850003 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133523580381184 FDeviceStatsMonitor.on_train_batch_en“½Zid/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10478107 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32903780 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29328112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1933383524719 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196227072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130208070426347 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1940372105728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130208858114319 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29327566 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133523580381184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1899 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41916854 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32903780Ìk˜ëý&Ò÷M REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18850003 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133523580381184 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10478107 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32903780 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29328112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1933383524719 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196227072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130208070426347 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1940372105728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130208858114319 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29327566 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133523580381184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1899 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41916854 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32903780 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133522789721088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74843676 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41939894 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1939599448576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1940372105728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74843676 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220167168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146215307500544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74820634 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32903782 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚146215331440640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884899429690204e+09 _step‚153 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32903782 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131583208275456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1939599448576 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74820634 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148240642614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18850005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1932612876107 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2025335113728 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992™b˜Š 2Ó- _wandbruntime‚2042  _runtime‚2042Š.[ÔÕ /loss/FRt2/detection_loss‚2.2588765073466943 1loss/SRqqt1/detection_loss‚2.7323045272093553 0loss/SRqqt4/detection_loss‚1.122011206396248 0loss/FRt3/assignment_loss‚2.9438034105300903 -loss/FRt4/detection_loss‚2.00459906991039 /loss/FBt2/detection_loss‚2.5328074523380826 trainer/global_step‚1899 2loss/SRqqt1/assignment_loss‚1.7912312940909312 1loss/FBt2/assignment_loss‚0.07946866797283292 0loss/SRqqt2/detection_loss‚2.546877929457912 &loss/total_loss‚44.487129899778125 $ _timestamp‚1.788489942972404e+09 2loss/SRqqt3/assignment_loss‚1.6539238722684482 2loss/SRqqt4/assignment_loss‚1.0002093442540239 0loss/FBt3/assignment_loss‚0.2985593391931616 0loss/FRt4/assignment_loss‚1.2540342041424342 1loss/FBt1/assignment_loss‚0.13216645387001336 .loss/FRt3/detection_loss‚2.311584532856941 1loss/SRqqt3/detection_loss‚1.6666826605796814 2loss/SRqqt2/assignment_loss‚2.1687398276947163 /loss/FBt1/detection_loss‚3.4773043245077133 /loss/FRt2/assignment_loss‚3.718832692584476 1loss/FBt4/assignment_loss‚0.13574054390791038 .loss/FBt3/detection_loss‚2.505401259660721 /loss/FBt4/detection_loss‚0.5886212608254537 _runtime‚2042.612581026 0loss/FRt1/assignment_loss‚3.0036769906679788 /loss/FRt1/detection_loss‚2.5596725275119145 epoch‚1 _step‚154\Øôæ`ÕÚ /loss/FRt2/detection_loss‚2.2588765073466943 1loss/SRqqt1/detection_loss‚2.7323045272093553 0loss/SRqqt4/detection_loss‚1.122011206396248 0loss/FRt3/assignment_loss‚2.9438034105300903 -loss/FRt4/detection_loss‚2.00459906991039 /loss/FBt2/detection_loss‚2.5328074523380826 trainer/global_step‚1899 2loss/SRqqt1/assignment_loss‚1.7912312940909312 1loss/FBt2/assignment_loss‚0.07946866797283292 0loss/SRqqt2/detection_loss‚2.546877929457912 &loss/total_loss‚44.487129899778125 $ _timestamp‚1.788489942972404e+09 2loss/SRqqt3/assignment_loss‚1.6539238722684482 2loss/SRqqt4/assignment_loss‚1.0002093442540239 0loss/FBt3/assignment_loss‚0.2985593391931616 0loss/FRt4/assignment_loss‚1.2540342041424342 1loss/FBt1/assignment_loss‚0.13216645387001336 .loss/FRt3/detection_loss‚2.311584532856941 1loss/SRqqt3/detection_loss‚1.6666826605796814 2loss/SRqqt2/assignment_loss‚2.1687398276947163 /loss/FBt1/detection_loss‚3.4773043245077133 /loss/FRt2/assignment_loss‚3.718832692584476 1loss/FBt4/assignment_loss‚0.13574054390791038 .loss/FBt3/detection_loss‚2.505401259660721 /loss/FBt4/detection_loss‚0.5886212608254537 _runtime‚2042.612581026 0loss/FRt1/assignment_loss‚3.0036769906679788 /loss/FRt1/detection_loss‚2.5596725275119145 epoch‚1 _step‚154šÐbf»’Ö‚(:‡ ãáèÔú•® gpu.0.gpu‚48.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.233' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ5ðrׂ(:ç ãáèÔýñš¯ disk.sda.in‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚47332.349609375 disk.dm-0.in‚ 5702.61328125! disk.dm-0.out‚47332.349609375 disk.sda3.in‚0 disk.dm-1.out‚ 61.78515625 disk.sda.out‚ 61.78515625 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 115.48046875 cpu‚24.957178386997892 proc.cpu.threads‚17 network.recv‚688415) disk./.usagePercent‚78.64793942198169" disk./.usageGB‚1171.9111328125! disk.nvme0n1.in‚ 5702.61328125$ disk.nvme0n1.out‚47332.349609375$ memory_percent‚99.29516315460205 network.sent‚ 3.414654e+06 disk.drbd1260.out‚ 0.28515625 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 5702.61328125# proc.memory.rssMB‚ 2927.16796875* proc.memory.percent‚17.866015434265137 disk.sda3.out‚ 61.78515625 disk.dm-1.in‚0 disk.drbd1260.in‚0z­ì6’Ø‚(:‡ òáèԘɌ­ gpu.0.gpu‚48.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.754' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3508 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øÉv rÙ‚(:ç òáèÔ©Òê®% disk./.usageGB‚1172.0463562011719 disk.sda.out‚ 62.13671875 disk.sda2.out‚0$ memory_percent‚99.29518699645996' proc.memory.availableMB‚ 115.4765625* proc.memory.percent‚17.866015434265137 cpu‚24.9579513193461 network.sent‚ 3.522373e+06) disk./.usagePercent‚78.65701437705543& disk.nvme0c0n1.out‚47508.775390625$ disk.nvme0n1.out‚47508.775390625 disk.dm-0.in‚ 5702.61328125 disk.dm-1.in‚0 disk.dm-1.out‚ 62.13671875 disk.drbd1260.out‚ 0.28515625# disk.nvme0c0n1.in‚ 5702.61328125! disk.dm-0.out‚47508.775390625 disk.sda3.in‚0 disk.sda3.out‚ 62.13671875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 5702.61328125 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2927.16796875 proc.cpu.threads‚17 network.recv‚693866ÀôØÀ’Ú‚(:‡ âèÔ—þÑ° gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚128.141' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.6282 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤IígÛ‚(:Ü âèÔ™±€³ disk.dm-1.out‚62.5625# disk.dm-0.out‚47853.89501953125* proc.memory.percent‚17.866015434265137 cpu‚24.95872111259286 disk.drbd1260.out‚ 0.2890625 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 5830.9453125 proc.cpu.threads‚17 disk.sda1.in‚0 network.recv‚698494% disk./.usageGB‚1172.0695152282715 disk.sda3.in‚0 disk.sda3.out‚62.5625 disk.dm-1.in‚0 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚47853.89501953125) disk./.usagePercent‚78.65856859879581 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 5830.9453125 disk.nvme0n1.in‚ 5830.9453125& disk.nvme0n1.out‚47853.89501953125$ memory_percent‚99.29370880126953% proc.memory.availableMB‚ 115.71875# proc.memory.rssMB‚ 2927.16796875 disk.sda.out‚62.5625 network.sent‚ 3.528555e+06ÖRYw2Ü- _wandbruntime‚2087  _runtime‚2087Í7¼VÆÝÀ & lr-AdamW/pg1‚0.0007832353025302059 & lr-AdamW/pg2‚0.0007832353025302059 trainer/global_step‚1949 $ _timestamp‚1.788489988151597e+09 _runtime‚2087.791877954 _step‚155~·EMËÞÅ & lr-AdamW/pg1‚0.0007832353025302059 & lr-AdamW/pg2‚0.0007832353025302059 trainer/global_step‚1949 $ _timestamp‚1.788489988151597e+09 _runtime‚2087.791877954 _step‚155›¢%À=2ß- _wandbruntime‚2087  _runtime‚2087 ÷.í'àçO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19322757 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚151984371856896 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33730606 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30061876 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134900487299584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136889333702656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1989619046912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10738584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1988846403072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149907376296960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 trainer/global_step‚1949 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134900505302528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2076971619840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33730608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1988846403072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33730606 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134900487299584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33730608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43007059 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42984045 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 181560320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1982452969741 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10739119 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚151984166356480 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149907400237056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42984045 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 _step‚156 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131512729941840 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1981682321337 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131512712902480 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43007059 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 % _timestamp‚1.7884899881524992e+09 _runtime‚ 2087.79292103 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134900505302528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚136890124349440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76737667 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19322755 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992æè½òSáìO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30061339 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133494395223817 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚136890124349440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136889333702656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1989619046912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10738584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1988846403072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149907376296960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 trainer/global_step‚1949 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.largÇÔùÙŸ!e_pool.allocated‚134900505302528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2076971619840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33730608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1988846403072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33730606 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134900487299584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33730608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43007059 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42984045 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 181560320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1982452969741 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10739119 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚151984166356480 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149907400237056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42984045 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 _step‚156 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131512729941840 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1981682321337 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131512712902480 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43007059 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 % _timestamp‚1.7884899881524992e+09 _runtime‚ 2087.79292103 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134900505302528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚136890124349440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76737667 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19322755 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19322757 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚151984371856896 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33730606 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30061876 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134900487299584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43005756 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152060782734848 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149982724639744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10744344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _runtime‚2088.658828923 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134968187238912 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43028796 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76776278 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33747480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152060564664832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136958829047808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1990623805952 % _timestamp‚1.7884899890177035e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 _step‚157 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33747480 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2077840025088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30076237 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43005756 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134968205241856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134968187238912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1989851148800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76753236 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19332405 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43028796 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131578779338240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19332403 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76753236 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1989851148800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33747482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149982748579840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2077840025088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30076237 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43005756 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134968205241856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134968187238912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1989851148800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76753236 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19332405 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43028796 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131578779338240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19332403 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76753236 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1989851148800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33747482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149982748579840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43005756 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152060782734848 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149982724639744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10744344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _runtime‚2088.658828923 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134968187238912 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43028796 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76776278 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33747480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152060564664832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136958829047808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1990623805952 % _timestamp‚1.7884899890177035e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 _step‚157­wÇ2å- _wandbruntime‚2088  _runtime‚2088)®Ÿß[æÕ &loss/total_loss‚45.388498942946825 epoch‚1 0loss/FBt4/assignment_loss‚0.0644944285344881 .loss/FRt3/detection_loss‚2.677216912017149 1loss/SRqqt3/detection_loss‚1.7897597887213268 1loss/SRqqt2/detection_loss‚2.2751268870902783 .loss/FBt1/detection_loss‚2.984095660122958 _runtime‚ 2088.66009506 _step‚158 1loss/FBt2/assignment_loss‚0.30191077185172616 .loss/FRt1/detection_loss‚2.455757550473483 1loss/SRqqt4/detection_loss‚0.9485501513179195 /loss/FBt4/detection_loss‚0.6701155557212505 trainer/global_step‚1949 % _timestamp‚1.7884899890199172e+09 0loss/FRt1/assignment_loss‚3.6702992994830295 /loss/FRt4/assignment_loss‚1.629126722259181 2loss/SRqqt4/assignment_loss‚1.1894849566742778 /loss/FRt4/detection_loss‚1.7933076512334603 0loss/SRqqt1/detection_loss‚2.832785270535029 /loss/FRt3/assignment_loss‚3.572423540494021 2loss/SRqqt3/assignment_loss‚1.8660111507726094 /loss/FRt2/detection_loss‚2.1745186345917844 /loss/FBt3/detection_loss‚2.6019877195358276 /loss/FRt2/assignment_loss‚4.265407839843205 2loss/SRqqt1/assignment_loss‚1.4221835679446275 2loss/SRqqt2/assignment_loss‚1.9794741607073583 1loss/FBt1/assignment_loss‚0.17329216942248007 1loss/FBt3/assignment_loss‚0.06723177290820916 /loss/FBt2/detection_loss‚1.9839367806911468`ôG4`çÚ /loss/FRt2/assignment_loss‚4.265407839843205 2loss/SRqqt1/assignment_loss‚1.4221835679446275 2loss/SRqqt2/assignment_loss‚1.9794741607073583 1loss/FBt1/assignment_loss‚0.17329216942248007 1loss/FBt3/assignment_loss‚0.06723177290820916 /loss/FBt2/detection_loss‚1.9839367806911468 &loss/total_loss‚45.388498942946825 epoch‚1 0loss/FBt4/assignment_loss‚0.0644944285344881 .loss/FRt3/detection_loss‚2.677216912017149 1loss/SRqqt3/detection_loss‚1.7897597887213268 1loss/SRqqt2/detection_loss‚2.2751268870902783 .loss/FBt1/detection_loss‚2.984095660122958 _runtime‚ 2088.66009506 _step‚158 1loss/FBt2/assignment_loss‚0.30191077185172616 .loss/FRt1/detection_loss‚2.455757550473483 1loss/SRqqt4/detection_loss‚0.9485501513179195 /loss/FBt4/detection_loss‚0.6701155557212505 trainer/global_step‚1949 % _timestamp‚1.7884899890199172e+09 0loss/FRt1/assignment_loss‚3.6702992994830295 /loss/FRt4/assignment_loss‚1.629126722259181 2loss/SRqqt4/assignment_loss‚1.1894849566742778 /loss/FRt4/detection_loss‚1.7933076512334603 0loss/SRqqt1/detection_loss‚2.832785270535029 /loss/FRt3/assignment_loss‚3.572423540494021 2loss/SRqqt3/assignment_loss‚1.8660111507726094 /loss/FRt2/detection_loss‚2.1745186345917844 /loss/FBt3/detection_loss‚2.6019877195358276žnã½è‚(:’ âèÔ֨ͯ gpu.0.gpu‚28.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.554' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.110800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àe né‚(:ã âèÔг´± disk.sda1.in‚0( disk.nvme0c0n1.out‚48146.56201171875# disk.dm-0.out‚48146.56201171875# proc.memory.rssMB‚ 2927.16796875 proc.cpu.threads‚17 network.sent‚ 3.5344e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 62.953125 disk.sda1.out‚0* proc.memory.percent‚17.866015434265137 network.recv‚702691 disk.sda3.out‚ 62.953125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2890625 disk.nvme0n1.in‚ 5830.9453125& disk.nvme0n1.out‚48146.56201171875 disk.dm-0.in‚ 5830.9453125$ memory_percent‚99.29547309875488 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 5830.9453125' proc.memory.availableMB‚ 115.4296875 cpu‚24.959730210370125) disk./.usagePercent‚78.65887196772388% disk./.usageGB‚1172.0740356445312 disk.sda.out‚ 62.953125WžK†ê‚(:… ŸâèÔö芰 gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚127.32' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.464 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýž³që‚(:æ ŸâèÔ…Ž×°% disk./.usageGB‚1172.0740432739258 disk.dm-1.out‚ 63.45703125 disk.sda.out‚ 63.45703125 disk.sda2.out‚0 disk.dm-0.in‚ 5830.9453125 cpu‚24.960712862153 proc.cpu.threads‚17 network.recv‚708083 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 5830.9453125 disk.sda3.out‚ 63.45703125$ memory_percent‚99.29547309875488 network.sent‚ 3.643821e+06) disk./.usagePercent‚78.65887247973895 disk.dm-1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2927.16796875* proc.memory.percent‚17.866015434265137 disk.drbd1260.out‚ 0.29296875 disk.sda.in‚0" disk.nvme0c0n1.in‚ 5830.9453125' disk.nvme0c0n1.out‚48410.8427734375% disk.nvme0n1.out‚48410.8427734375" disk.dm-0.out‚48410.8427734375 disk.sda3.in‚0' proc.memory.availableMB‚ 115.4296875‘R×’ì‚(:‡ ®âèÔã÷ï® gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.8412 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œàáÓtí‚(:é ®âèÔ–Ðü²) disk./.usagePercent‚78.66306025098405% disk./.usageGB‚1172.1364440917969 disk.sda3.in‚0 disk.dm-1.out‚ 63.83203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 5830.9453125 disk.dm-0.in‚ 5830.9453125 network.sent‚ 3.648981e+06 disk.drbd1260.out‚ 0.29296875" disk.nvme0c0n1.in‚ 5830.9453125" disk.dm-0.out‚48752.5966796875$ memory_percent‚99.29568767547607 cpu‚24.96159738369739 proc.cpu.threads‚17 network.recv‚711795 disk.sda.in‚0 disk.sda.out‚ 63.83203125 disk.sda1.in‚0% disk.nvme0n1.out‚48752.5966796875* proc.memory.percent‚17.866015434265137 disk.sda3.out‚ 63.83203125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚48752.5966796875( proc.memory.availableMB‚ 115.39453125# proc.memory.rssMB‚ 2927.16796875?»y2î- _wandbruntime‚2136  _runtime‚2136 ²Å"ÇïÁ _step‚159 & lr-AdamW/pg1‚0.0007819825052734656 & lr-AdamW/pg2‚0.0007819825052734656 trainer/global_step‚1999 % _timestamp‚1.7884900372877007e+09 _runtime‚2136.928140172…ns#ÌðÆ & lr-AdamW/pg1‚0.0007819825052734656 & lr-AdamW/pg2‚0.0007819825052734656 trainer/global_step‚1999 % _timestamp‚1.7884900372877007e+09 _runtime‚2136.928140172 _step‚159Ÿ–O‰Z2ñ- _wandbruntime‚2136  _runtime‚2136òþÁî'òèO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2039098103296 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30810911 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138285484265984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138285484265984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44072947 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153674793436160 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 _step‚160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34574306 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204628992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140325373016064 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚140324582369280 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19805157 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44095961 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153674817376256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78670269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚136847787596925 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134816051729840 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2031752906445 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19805155 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34574308 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34574308 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155804338788864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34574306 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30810353 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140324582369280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚138285502268928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228569088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489ÃRluó'óíO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44095961 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153674817376256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78670269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚136847787596925 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134816051729840 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2031752906445 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19805155 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34574308 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34574308 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155804338788864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34574306 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30810353 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140324582369280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚138285502268928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228569088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2039098103296 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30810911 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138285484265984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138285484265984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44072947 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153674793436160 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 _step‚160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34574306 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204628992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140325373016064 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚140324582369280 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19805157 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ÚÝB—2ô- _wandbruntime‚2137  _runtime‚2137ò m}›#õòM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44094658 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34591182 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2040875506176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140394077714432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚138353184205312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚138353184205312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2130558088704 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 trainer/global_step‚1999 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138353202208256 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2040875506176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155880723807744 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134882101126240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34591182 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140394077714432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11010459 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19814805 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78685838 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78685838 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30825801 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134882118165600 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78708880 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19814803 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34591180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30825262 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44094658 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2040102849024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44117698 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34591180 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224361472 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2033524694935 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155880499446272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200421376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2130357667328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140393287054336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 % _timestamp‚1.7884900382499835e+09 _runtime‚2137.891076802 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153750141778944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78708880 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÆÔü%ý&ö÷M WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134882101126240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34591182 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140394077714432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11010459 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19814805 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78685838 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78685838 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30825801 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134882118165600 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78708880 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19814803 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34591180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30825262 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44094658 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2040102849024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44117698 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34591180 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224361472 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2033524694935 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155880499446272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200421376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2130357667328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140393287054336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 % _timestamp‚1.7884900382499835e+09 _runtime‚2137.891076802 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153750141778944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78708880 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44094658 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34591182 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2040875506176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140394077714432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚138353184205312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚138353184205312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2130558088704 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 trainer/global_step‚1999 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138353202208256 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2040875506176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155880723807744 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992¡®ªÛÝ2÷- _wandbruntime‚2137  _runtime‚2137ýT¤½]ø× /loss/FRt1/assignment_loss‚4.069777953624725 0loss/FRt4/assignment_loss‚1.3826789899305865 1loss/SRqqt1/assignment_loss‚2.187315486371517 1loss/SRqqt4/detection_loss‚1.0104175543083864 epoch‚1 -loss/FBt2/detection_loss‚4.28739932179451 % _timestamp‚1.7884900382527888e+09 /loss/FRt2/detection_loss‚2.2247713455494416 1loss/SRqqt3/assignment_loss‚1.924468574339919 2loss/SRqqt4/assignment_loss‚0.9838283128543374 1loss/SRqqt1/detection_loss‚2.1378362702160345 /loss/FBt4/detection_loss‚0.6066036464206959 _step‚162 1loss/SRqqt2/assignment_loss‚2.186431823550044 2loss/FBt2/assignment_loss‚0.007025234650427592 1loss/FBt4/assignment_loss‚0.08792176099528108 /loss/FBt3/detection_loss‚2.3616190378864608 1loss/FBt3/assignment_loss‚0.11049432453516299 .loss/FRt1/detection_loss‚2.411520246267319 1loss/SRqqt3/detection_loss‚1.5305184191375067 trainer/global_step‚1999 0loss/FRt2/assignment_loss‚3.8069595580405378 /loss/FRt4/detection_loss‚1.4078044201840054 0loss/SRqqt2/detection_loss‚2.280171722979159 0loss/FRt3/assignment_loss‚3.1314427974549206 _runtime‚2137.893053448 1loss/FBt1/assignment_loss‚0.08581843654004236 /loss/FRt3/detection_loss‚2.3031035394153814 /loss/FBt1/detection_loss‚3.0342032512029014 &loss/total_loss‚45.560132028249306ÓûÏUbùÜ 2loss/FBt2/assignment_loss‚0.007025234650427592 1loss/FBt4/assignment_loss‚0.08792176099528108 /loss/FBt3/detection_loss‚2.3616190378864608 1loss/FBt3/assignment_loss‚0.11049432453516299 .loss/FRt1/detection_loss‚2.411520246267319 1loss/SRqqt3/detection_loss‚1.5305184191375067 trainer/global_step‚1999 0loss/FRt2/assignment_loss‚3.8069595580405378 /loss/FRt4/detection_loss‚1.4078044201840054 0loss/SRqqt2/detection_loss‚2.280171722979159 0loss/FRt3/assignment_loss‚3.1314427974549206 _runtime‚2137.893053448 1loss/FBt1/assignment_loss‚0.08581843654004236 /loss/FRt3/detection_loss‚2.3031035394153814 /loss/FBt1/detection_loss‚3.0342032512029014 &loss/total_loss‚45.560132028249306 /loss/FRt1/assignment_loss‚4.069777953624725 0loss/FRt4/assignment_loss‚1.3826789899305865 1loss/SRqqt1/assignment_loss‚2.187315486371517 1loss/SRqqt4/detection_loss‚1.0104175543083864 epoch‚1 -loss/FBt2/detection_loss‚4.28739932179451 % _timestamp‚1.7884900382527888e+09 /loss/FRt2/detection_loss‚2.2247713455494416 1loss/SRqqt3/assignment_loss‚1.924468574339919 2loss/SRqqt4/assignment_loss‚0.9838283128543374 1loss/SRqqt1/detection_loss‚2.1378362702160345 /loss/FBt4/detection_loss‚0.6066036464206959 _step‚162 1loss/SRqqt2/assignment_loss‚2.186431823550044¢X“Àóú‚(:’ ½âèÔÁßÈ® gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚125.292' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.058400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&rr&qû‚(:æ ½âèÔØÚê¯ disk.dm-1.out‚ 64.17578125' proc.memory.availableMB‚ 114.8671875 proc.cpu.threads‚17 network.recv‚716460 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 5973.9609375 network.sent‚ 3.653261e+06) disk./.usagePercent‚78.66725467842505 disk.sda.out‚ 64.17578125' disk.nvme0c0n1.out‚48807.4228515625$ memory_percent‚99.29890632629395# proc.memory.rssMB‚ 2927.16796875* proc.memory.percent‚17.866015434265137% disk./.usageGB‚1172.1989440917969 disk.dm-0.in‚ 5973.9609375" disk.dm-0.out‚48807.4228515625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.296875 disk.nvme0n1.in‚ 5973.9609375% disk.nvme0n1.out‚48807.4228515625 cpu‚24.96249073415025 disk.sda3.out‚ 64.17578125 disk.dm-1.in‚0§UÎÂ’ü‚(:‡ ÌâèÔÝóˆ± gpu.0.gpu‚37.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0604 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+šçäpý‚(:å ÌâèÔÖ˜æµ) disk./.usagePercent‚78.67405910267762" disk.dm-0.out‚48980.9150390625 disk.sda3.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 115.08984375# proc.memory.rssMB‚ 2927.16796875 proc.cpu.threads‚17 network.sent‚ 3.763567e+06 network.recv‚722988 disk.dm-1.in‚0 disk.dm-1.out‚ 64.57421875 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚48980.9150390625# disk./.usageGB‚1172.30033493042 disk.nvme0n1.in‚ 5973.9609375% disk.nvme0n1.out‚48980.9150390625 disk.sda3.out‚ 64.57421875 disk.sda.in‚0 disk.sda.out‚ 64.57421875 disk.sda1.out‚0$ memory_percent‚99.29754734039307 disk.dm-0.in‚ 5973.9609375 disk.drbd1260.out‚0.296875" disk.nvme0c0n1.in‚ 5973.9609375* proc.memory.percent‚17.866015434265137 cpu‚24.96349642851079£þ Óþ‚(:’ ÛâèÔÍÀŒ² gpu.0.gpu‚36.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.449' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.689800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FëþSjÿ‚(:ß ÛâèÔ’懼 disk.dm-1.out‚ 65.0078125 disk.drbd1260.out‚ 0.30078125 disk.sda.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 5973.9609375( proc.memory.availableMB‚ 115.23046875# proc.memory.rssMB‚ 2927.16796875 network.sent‚ 3.766795e+06$ disk./.usageGB‚1172.157859802246 disk.sda3.in‚0 disk.sda2.out‚0 cpu‚24.964706918804747 network.recv‚726396# disk.nvme0n1.out‚49348.68359375 disk.dm-0.in‚ 5973.9609375 disk.dm-0.out‚49348.68359375 disk.sda3.out‚ 65.0078125% disk.nvme0c0n1.out‚49348.68359375# memory_percent‚99.2966890335083) disk./.usagePercent‚78.66449747728129 disk.nvme0n1.in‚ 5973.9609375 disk.drbd1260.in‚0 disk.sda.out‚ 65.0078125 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.866015434265137 proc.cpu.threads‚17 disk.dm-1.in‚0 cì2€- _wandbruntime‚2187  _runtime‚2187­ÏÐÅ¿ % lr-AdamW/pg1‚0.000780649295068649 % lr-AdamW/pg2‚0.000780649295068649 trainer/global_step‚2049 % _timestamp‚1.7884900876966655e+09 _runtime‚2187.336958368 _step‚163±†ÂÊ‚Ä % lr-AdamW/pg1‚0.000780649295068649 % lr-AdamW/pg2‚0.000780649295068649 trainer/global_step‚2049 % _timestamp‚1.7884900876966655e+09 _runtime‚2187.336958368 _step‚163£ypú¦2ƒ- _wandbruntime‚2187  _runtime‚2187<«g î'„èO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45161849 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20287557 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 % _timestamp‚1.7884900876976657e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚143759831035904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138119356478480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31560616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138119373517840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2090122447360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31560091 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚159624147456512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143760621682688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20287555 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35418008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11273059 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141670481232384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45184863 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2089349803520 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚140201967657797 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚140201179970033 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2089349803520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213889024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45184863 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 _step‚164 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143760621682688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _runtime‚2187.338099168 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80579855 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80602871 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45161849 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2090122447360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11272536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157442234515456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2181936881152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143759831035904 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2182126830080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚159624361345536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992K2AF…íO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20287557 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 % _timestamp‚1.7884900876976657e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚143759831035904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138119356478480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31560616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138119373517840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2090122447360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31560091 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚159624147456512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143760621682688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20287555 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35418008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11273059 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141670481232384 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45184863 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_po=uSÖ­ol.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2089349803520 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚140201967657797 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚140201179970033 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2089349803520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213889024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45184863 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 _step‚164 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143760621682688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _runtime‚2187.338099168 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80579855 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80602871 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45161849 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2090122447360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11272536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157442234515456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2181936881152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143759831035904 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2182126830080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚159624361345536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45161849 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360¤U„q•2†- _wandbruntime‚2188  _runtime‚2188Žguø&‡òM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45206600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159700766500352 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2082824631431 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80641482 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2091127206400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2090354549248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141738199174656 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45183560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159700546333184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35434882 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚141738199174656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35434880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 % _timestamp‚1.7884900885426922e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141738181171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 trainer/global_step‚2049 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31575019 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20297205 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2182987415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚2188.183646017 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35434882 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138185422914240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45206600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20297203 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11278353 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141738181171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196227072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220167168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚140268247545671 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35434880 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31575558 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80641482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11277816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2083595280043 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45183560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138185439953600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80618440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2183183642112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80641482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11277816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2083595280043 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45183560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138185439953600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80618440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2183183642112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31575019 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20297205 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2182987415040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚2188.183646017 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35434882 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138185422914240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45206600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20297203 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11278353 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141738181171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196227072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220167168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚140268247545671 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35434880 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31575558¥°(ü.2‰- _wandbruntime‚2188  _runtime‚2188Mî«^ŠØ 2loss/SRqqt1/assignment_loss‚2.0396062370170567 1loss/FBt1/assignment_loss‚0.21029649462434463 1loss/FBt4/assignment_loss‚0.10451106029064769 1loss/SRqqt3/detection_loss‚1.4337766530258316 % _timestamp‚1.7884900885452368e+09 _runtime‚2188.185403568 /loss/FRt3/detection_loss‚2.3763213411290596 epoch‚1 /loss/FBt3/detection_loss‚1.7250880140524645 trainer/global_step‚2049 0loss/FRt1/assignment_loss‚3.3569752899083225 /loss/FRt2/assignment_loss‚3.514695182442665 0loss/FRt3/assignment_loss‚2.7364491084788707 2loss/SRqqt2/assignment_loss‚2.1870786355514276 2loss/SRqqt4/assignment_loss‚0.8430342912429669 &loss/total_loss‚44.367941935066966 0loss/FRt4/assignment_loss‚1.7376046043175917 .loss/FRt2/detection_loss‚2.640077957510948 0loss/SRqqt2/detection_loss‚2.325584886105437 .loss/FBt1/detection_loss‚3.086449258029461 /loss/FBt4/detection_loss‚0.6304880759819562 _step‚166 2loss/SRqqt3/assignment_loss‚1.8274603473288673 1loss/SRqqt1/detection_loss‚2.4372876286506653 0loss/SRqqt4/detection_loss‚1.114746379681298 /loss/FBt2/detection_loss‚3.3359290758768716 1loss/FBt2/assignment_loss‚0.48222847500195104 0loss/FBt3/assignment_loss‚0.1377475688663813 /loss/FRt1/detection_loss‚2.6686876206235453 /loss/FRt4/detection_loss‚1.4158177493283381Z*©¹c‹Ý /loss/FRt4/detection_loss‚1.4158177493283381 2loss/SRqqt1/assignment_loss‚2.0396062370170567 1loss/FBt1/assignment_loss‚0.21029649462434463 1loss/FBt4/assignment_loss‚0.10451106029064769 1loss/SRqqt3/detection_loss‚1.4337766530258316 % _timestamp‚1.7884900885452368e+09 _runtime‚2188.185403568 /loss/FRt3/detection_loss‚2.3763213411290596 epoch‚1 /loss/FBt3/detection_loss‚1.7250880140524645 trainer/global_step‚2049 0loss/FRt1/assignment_loss‚3.3569752899083225 /loss/FRt2/assignment_loss‚3.514695182442665 0loss/FRt3/assignment_loss‚2.7364491084788707 2loss/SRqqt2/assignment_loss‚2.1870786355514276 2loss/SRqqt4/assignment_loss‚0.8430342912429669 &loss/total_loss‚44.367941935066966 0loss/FRt4/assignment_loss‚1.7376046043175917 .loss/FRt2/detection_loss‚2.640077957510948 0loss/SRqqt2/detection_loss‚2.325584886105437 .loss/FBt1/detection_loss‚3.086449258029461 /loss/FBt4/detection_loss‚0.6304880759819562 _step‚166 2loss/SRqqt3/assignment_loss‚1.8274603473288673 1loss/SRqqt1/detection_loss‚2.4372876286506653 0loss/SRqqt4/detection_loss‚1.114746379681298 /loss/FBt2/detection_loss‚3.3359290758768716 1loss/FBt2/assignment_loss‚0.48222847500195104 0loss/FBt3/assignment_loss‚0.1377475688663813 /loss/FRt1/detection_loss‚2.6686876206235453¦^ɇ’Œ‚(:‡ êâèÔè½ú® gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.981' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9962 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªàÙ_‚(:Ô êâèÔ÷þ÷¯$ memory_percent‚99.29986000061035 disk.dm-0.in‚ 5973.9609375 disk.dm-0.out‚ 49786.7265625 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚65.4375' proc.memory.availableMB‚ 114.7109375 cpu‚24.965434564724458 proc.cpu.threads‚17) disk./.usagePercent‚78.68119352064183 disk.sda2.in‚0# proc.memory.rssMB‚ 2927.16796875* proc.memory.percent‚17.866015434265137% disk./.usageGB‚1172.4066429138184 disk.nvme0n1.in‚ 5973.9609375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.30078125 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 5973.9609375 network.sent‚ 3.771749e+06 network.recv‚729796" disk.nvme0n1.out‚ 49786.7265625 disk.sda3.out‚65.4375 disk.dm-1.out‚65.4375 disk.sda2.out‚0$ disk.nvme0c0n1.out‚ 49786.72656255Ñ­’Ž‚(:‡ ùâèÔÎЯ® gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.836' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.1672 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·i!;o‚(:ä ùâèÔ•µç²# disk.nvme0c0n1.in‚ 6116.65234375( disk.nvme0c0n1.out‚50310.18994140625! disk.nvme0n1.in‚ 6116.65234375( proc.memory.availableMB‚ 114.71484375* proc.memory.percent‚17.866015434265137 network.recv‚735668$ disk./.usageGB‚1172.530559539795& disk.nvme0n1.out‚50310.18994140625 disk.sda3.in‚0 disk.sda3.out‚65.84375 disk.drbd1260.in‚0 disk.sda.out‚65.84375 disk.sda1.in‚0 network.sent‚ 3.879589e+06 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.3046875 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚24.96625399830749 proc.cpu.threads‚17) disk./.usagePercent‚78.68950966938408 disk.dm-0.in‚ 6116.65234375# disk.dm-0.out‚50310.18994140625 disk.dm-1.out‚65.84375 disk.sda1.out‚0$ memory_percent‚99.29983615875244# proc.memory.rssMB‚ 2927.16796875Žç}›œ‚(:‘ ˆãèÔÚ“ª­ gpu.0.gpu‚2.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.699' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.339800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145055478198784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2139601503744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163444207618560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141422695305840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185754624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145055478198784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2140374147584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32310144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32309611 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143555360030905 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143554572343141 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145055496201728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36261708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147195870349312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145055496201728 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147195079702528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209694720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36261706 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2140374147584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2234765658624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82535473 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46273765 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11540187 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46250751 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11539656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2236.770103614 _step‚168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2131894076661 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 7906467842âó'™íO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2131894076661 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145055478198784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2139601503744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163444207618560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141422695305840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185754624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145055478198784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2140374147584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32310144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32309611 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143555360030905 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143554572343141 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145055496201728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36261708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147195870349312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145055496201728 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147195079702528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209694720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36261706 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2140374147584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2234765658624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82535473 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46273765 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11540187 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46250751 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11539656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2236.770103614 _step‚168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844¨N¯Ü–2š- _wandbruntime‚2237  _runtime‚2237tÐIø&›òM BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82574084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82551042 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209681408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 _runtime‚2237.745450581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20779603 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141488761741600 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143622427606751 trainer/global_step‚2099 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2141378906624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82551042 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46295502 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147264575047680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36278580 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32324637 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145123178138112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2132895216539 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36278580 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145123196141056 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161284976057344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141488744702240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36278582 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163520621332992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145123196141056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11545572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143621639918779 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 % _timestamp‚1.7884901381041956e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚147264575047680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163520831014400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2235831016960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145123178138112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20779605 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2140606249472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2235645275648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 _step‚169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82574084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46272462 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46295502 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ä¹Anœ÷M SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 % _timestamp‚1.7884901381041956e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚147264575047680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163520831014400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2235831016960 VHDeviceStatsMonitor.on_train_batch_end/requested_]”Åæ$bytes.small_pool.current‚ 770648612 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145123178138112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20779605 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2140606249472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2235645275648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 _step‚169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82574084 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46272462 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46295502 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82574084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82551042 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209681408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 _runtime‚2237.745450581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20779603 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141488761741600 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143622427606751 trainer/global_step‚2099 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2141378906624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82551042 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46295502 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147264575047680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36278580 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32324637 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145123178138112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2132895216539 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36278580 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145123196141056 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161284976057344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141488744702240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36278582 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163520621332992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145123196141056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11545572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143621639918779 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005©µn¶Œ2- _wandbruntime‚2237  _runtime‚2237¦æcMVžÐ 0loss/SRqqt4/detection_loss‚1.113727078067534 .loss/FBt2/detection_loss‚2.528025017844306 .loss/FBt3/detection_loss‚3.049717754125595 _step‚170 0loss/FBt2/assignment_loss‚0.6858890818095662 /loss/FRt2/detection_loss‚2.3876303214775887 1loss/SRqqt2/detection_loss‚2.2765876039480553 0loss/FBt1/assignment_loss‚0.5440959525294602 trainer/global_step‚2099 2loss/SRqqt1/assignment_loss‚2.0560274184681475 2loss/SRqqt4/assignment_loss‚0.9057119202263879 2loss/SRqqt2/assignment_loss‚2.1497737784416246 1loss/FBt3/assignment_loss‚0.36635065109779436 /loss/FBt1/detection_loss‚2.2895126144091287 /loss/FBt4/detection_loss‚0.6080939597212075 _runtime‚ 2237.74707983 /loss/FRt1/assignment_loss‚4.083409808252169 /loss/FRt1/detection_loss‚2.4020213491242863 0loss/FRt2/assignment_loss‚3.6266866828265942 /loss/FRt3/assignment_loss‚2.774561293423176 2loss/SRqqt3/assignment_loss‚1.5222681093275443 1loss/FBt4/assignment_loss‚0.21216814814515833 /loss/SRqqt3/detection_loss‚1.66300183824367 %loss/total_loss‚45.50974963456058 epoch‚1 % _timestamp‚1.7884901381067374e+09 0loss/FRt4/assignment_loss‚1.5547951328754426 -loss/FRt3/detection_loss‚2.46716019064188 /loss/FRt4/detection_loss‚1.7284316670894624 0loss/SRqqt1/detection_loss‚2.514102262444794 ò˜™[ŸÕ 0loss/FRt2/assignment_loss‚3.6266866828265942 /loss/FRt3/assignment_loss‚2.774561293423176 2loss/SRqqt3/assignment_loss‚1.5222681093275443 1loss/FBt4/assignment_loss‚0.21216814814515833 /loss/SRqqt3/detection_loss‚1.66300183824367 %loss/total_loss‚45.50974963456058 epoch‚1 % _timestamp‚1.7884901381067374e+09 0loss/FRt4/assignment_loss‚1.5547951328754426 -loss/FRt3/detection_loss‚2.46716019064188 /loss/FRt4/detection_loss‚1.7284316670894624 0loss/SRqqt1/detection_loss‚2.514102262444794 0loss/SRqqt4/detection_loss‚1.113727078067534 .loss/FBt2/detection_loss‚2.528025017844306 .loss/FBt3/detection_loss‚3.049717754125595 _step‚170 0loss/FBt2/assignment_loss‚0.6858890818095662 /loss/FRt2/detection_loss‚2.3876303214775887 1loss/SRqqt2/detection_loss‚2.2765876039480553 0loss/FBt1/assignment_loss‚0.5440959525294602 trainer/global_step‚2099 2loss/SRqqt1/assignment_loss‚2.0560274184681475 2loss/SRqqt4/assignment_loss‚0.9057119202263879 2loss/SRqqt2/assignment_loss‚2.1497737784416246 1loss/FBt3/assignment_loss‚0.36635065109779436 /loss/FBt1/detection_loss‚2.2895126144091287 /loss/FBt4/detection_loss‚0.6080939597212075 _runtime‚ 2237.74707983 /loss/FRt1/assignment_loss‚4.083409808252169 /loss/FRt1/detection_loss‚2.4020213491242863ªI™cF’ ‚(:‡ ¦ãèÔØËÏ° gpu.0.gpu‚43.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.9756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•ew¡‚(:ì ¦ãèÔÛÛ± disk.sda3.out‚ 67.3046875# disk.nvme0c0n1.in‚ 6116.65234375! disk.nvme0n1.in‚ 6116.65234375 disk.dm-0.in‚ 6116.65234375# disk.dm-0.out‚51126.41064453125( proc.memory.availableMB‚ 114.61328125* proc.memory.percent‚17.866015434265137 cpu‚24.968752386825013) disk./.usagePercent‚78.69967624058384 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.30859375 disk.sda.out‚ 67.3046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚51126.41064453125 network.sent‚ 3.895629e+06 network.recv‚746929% disk./.usageGB‚1172.6820487976074 disk.dm-1.out‚ 67.3046875 disk.sda.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚51126.41064453125# memory_percent‚99.3004560470581 disk.sda3.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2927.16796875 proc.cpu.threads‚17Þ¶°C‘¢‚(:† µãèԸ普 gpu.0.gpu‚9.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.882' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9764 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jTØm£‚(:â µãèÔì±ÿ® network.sent‚ 4.004479e+06# disk.dm-0.out‚51444.40966796875 disk.drbd1260.in‚0 disk.drbd1260.out‚0.3125 disk.sda1.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚51444.40966796875( disk./.usagePercent‚78.6981012822332 disk.dm-0.in‚ 6290.39453125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2927.16796875* proc.memory.percent‚17.866015434265137! disk.nvme0n1.in‚ 6290.39453125& disk.nvme0n1.out‚51444.40966796875 disk.sda3.out‚67.78125 disk.dm-1.out‚67.78125 disk.sda.out‚67.78125 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 6290.39453125 proc.cpu.threads‚17 network.recv‚753980% disk./.usageGB‚1172.6585807800293$ memory_percent‚99.30050373077393( proc.memory.availableMB‚ 114.60546875 cpu‚24.969545416968753ìv‘W’¤‚(:‡ Äãèԉį gpu.0.gpu‚48.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.531' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.7062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê[]øu¥‚(:ê ÄãèÔè葳* proc.memory.percent‚17.866015434265137 cpu‚24.97039014375837 network.recv‚758258$ disk./.usageGB‚1172.666519165039 disk.drbd1260.out‚0.3125 disk.sda2.out‚0( disk.nvme0c0n1.out‚51843.40576171875& disk.nvme0n1.out‚51843.40576171875# disk.dm-0.out‚51843.40576171875) disk./.usagePercent‚78.69863403391194 disk.drbd1260.in‚0 disk.sda.out‚ 68.19921875 disk.sda1.out‚0! disk.nvme0n1.in‚ 6290.39453125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 68.19921875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 6290.39453125# proc.memory.rssMB‚ 2927.16796875 network.sent‚ 4.010375e+06 disk.sda3.out‚ 68.19921875 disk.dm-0.in‚ 6290.39453125$ memory_percent‚99.30055141448975( proc.memory.availableMB‚ 114.59765625GDa¨2¦- _wandbruntime‚2287  _runtime‚2287v‘·Ç§Á & lr-AdamW/pg2‚0.0007777427638708577 trainer/global_step‚2149 % _timestamp‚1.7884901878790963e+09 _runtime‚2287.519622778 _step‚171 & lr-AdamW/pg1‚0.0007777427638708577ü—‘ÉÌ¨Æ & lr-AdamW/pg2‚0.0007777427638708577 trainer/global_step‚2149 % _timestamp‚1.7884901878790963e+09 _runtime‚2287.519622778 _step‚171 & lr-AdamW/pg1‚0.0007777427638708577«¢èì2©- _wandbruntime‚2287  _runtime‚2287ÍØhî'ªèO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150630328369152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164977044853760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2182735310173 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47362667 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148440493168128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2190625847808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚148440475165184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37105406 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2287.521011875 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144726000054480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 trainer/global_step‚2149 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37105406 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 _step‚172 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21252357 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215986176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2287189846016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2181964661769 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150631119015936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11806991 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2189853203968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150631119015936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚148440475165184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47339653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33058798 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164977068793856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167264450685952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0E5Q®+«íO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150630328369152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164977044853760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2182735310173 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47362667 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148440493168128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2190625847808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚148440475165184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37105406 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2287.521011875 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144726000054480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 trainer/global_step‚2149 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37105406 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 _step‚172 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21252357 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215986176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2287189846016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2181964661769 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150631119015936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11806991 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2189853203968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150631119015936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚148440475165184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47339653 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33058798 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164977068793856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167264450685952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0¬Žg 2¬- _wandbruntime‚2288  _runtime‚2288â¼÷&­ñM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47384404 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2190857949696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150699033054208 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2183736450259 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37122282 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33074163 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788490188738984e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47384404 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167340624257024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2288231060480 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150699823714304 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84506686 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2191630606848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2191630606848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37122280 _step‚173 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47361364 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚148508193107456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37122280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33073633 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146975032291887 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84506686 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37122282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21262005 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200421376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150699823714304 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84483644 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148508175104512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 _runtime‚2288.380029845 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚2149 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144792083529600-™”Bü&®öM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47361364 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚148508193107456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37122280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33073633 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146975032291887 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84506686 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37122282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21262005 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200421376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150699823714304 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84483644 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148508175104512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 _runtime‚2288.380029845 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚2149 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144792083529600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47384404 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2190857949696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150699033054208 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2183736450259 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37122282 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33074163 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788490188738984e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47384404 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167340624257024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2288231060480 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150699823714304 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84506686 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2191630606848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2191630606848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37122280 _step‚173­úkþj2¯- _wandbruntime‚2288  _runtime‚2288æ ¸ó]°× 1loss/SRqqt1/detection_loss‚1.8923016843341647 1loss/SRqqt3/detection_loss‚1.5769164864696674 /loss/FBt1/detection_loss‚6.1724677085876465 .loss/FBt2/detection_loss‚2.584482192993164 /loss/FBt3/detection_loss‚2.1766297679681044 trainer/global_step‚2149 /loss/FRt4/detection_loss‚1.5412000312385234 /loss/FBt4/detection_loss‚0.6211586066688423 _runtime‚2288.382175037 0loss/FRt2/assignment_loss‚3.6585655473172665 2loss/SRqqt2/assignment_loss‚1.5645816556232817 1loss/SRqqt3/assignment_loss‚1.517675742632084 1loss/FBt4/assignment_loss‚0.11620589433422536 /loss/FRt3/detection_loss‚2.2605749875307084 /loss/FRt3/assignment_loss‚2.864717814326286 1loss/FBt3/assignment_loss‚0.21755988106955415 .loss/FRt1/detection_loss‚2.476910852287945 $ _timestamp‚1.788490188742014e+09 2loss/SRqqt1/assignment_loss‚2.0871578937485102 _step‚174 /loss/FRt1/assignment_loss‚3.556733721180966 2loss/SRqqt4/assignment_loss‚1.0932204098781393 2loss/FBt1/assignment_loss‚0.024313723668456078 1loss/FBt2/assignment_loss‚0.03438400303712115 .loss/FRt2/detection_loss‚2.868068505078554 0loss/SRqqt2/detection_loss‚2.243758308536866 1loss/SRqqt4/detection_loss‚0.9511730370714384 &loss/total_loss‚45.478393706973534 epoch‚1 0loss/FRt4/assignment_loss‚1.3776352513920178­+ªb±Ü 2loss/SRqqt1/assignment_loss‚2.0871578937485102 _step‚174 /loss/FRt1/assignment_loss‚3.556733721180966 2loss/SRqqt4/assignment_loss‚1.0932204098781393 2loss/FBt1/assignment_loss‚0.024313723668456078 1loss/FBt2/assignment_loss‚0.03438400303712115 .loss/FRt2/detection_loss‚2.868068505078554 0loss/SRqqt2/detection_loss‚2.243758308536866 1loss/SRqqt4/detection_loss‚0.9511730370714384 &loss/total_loss‚45.478393706973534 epoch‚1 0loss/FRt4/assignment_loss‚1.3776352513920178 1loss/SRqqt1/detection_loss‚1.8923016843341647 1loss/SRqqt3/detection_loss‚1.5769164864696674 /loss/FBt1/detection_loss‚6.1724677085876465 .loss/FBt2/detection_loss‚2.584482192993164 /loss/FBt3/detection_loss‚2.1766297679681044 trainer/global_step‚2149 /loss/FRt4/detection_loss‚1.5412000312385234 /loss/FBt4/detection_loss‚0.6211586066688423 _runtime‚2288.382175037 0loss/FRt2/assignment_loss‚3.6585655473172665 2loss/SRqqt2/assignment_loss‚1.5645816556232817 1loss/SRqqt3/assignment_loss‚1.517675742632084 1loss/FBt4/assignment_loss‚0.11620589433422536 /loss/FRt3/detection_loss‚2.2605749875307084 /loss/FRt3/assignment_loss‚2.864717814326286 1loss/FBt3/assignment_loss‚0.21755988106955415 .loss/FRt1/detection_loss‚2.476910852287945 $ _timestamp‚1.788490188742014e+09®ˆ%¦Û’²‚(:‡ ÓãèÔ»‹¹° gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚165.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤oÙÉv³‚(:ë ÓãèÔÍ¥é² disk.sda1.out‚0( disk.nvme0c0n1.out‚51993.01611328125! disk.nvme0n1.in‚ 6290.39453125& disk.nvme0n1.out‚51993.01611328125 disk.dm-0.in‚ 6290.39453125# memory_percent‚99.2990255355835 disk.dm-1.in‚0 disk.sda.out‚ 68.5546875( proc.memory.availableMB‚ 114.84765625* proc.memory.percent‚17.866015434265137 cpu‚24.971215138910278% disk./.usageGB‚1172.3798141479492 disk.dm-1.out‚ 68.5546875 disk.drbd1260.out‚ 0.31640625 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 6290.39453125 disk.sda3.in‚0 disk.sda3.out‚ 68.5546875 network.sent‚ 4.015456e+06( disk./.usagePercent‚78.6793930196535 disk.sda.in‚0 disk.sda1.in‚0# disk.dm-0.out‚51993.01611328125# proc.memory.rssMB‚ 2927.16796875 proc.cpu.threads‚17 network.recv‚761659 disk.drbd1260.in‚0N’Ìk´‚(:’ âãèÔí›ñ­ gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.136' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.827199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý0vµ‚(:ë âãèÔܩݱ disk.drbd1260.out‚ 0.31640625 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚99.30212497711182 network.sent‚ 4.124924e+06( disk.nvme0c0n1.out‚52524.52783203125 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.62276901722271$ disk./.usageGB‚1171.536075592041& disk.nvme0n1.out‚52524.52783203125# disk.dm-0.out‚52524.52783203125 disk.drbd1260.in‚0 disk.sda.out‚ 69.0390625( proc.memory.availableMB‚ 114.33984375# proc.memory.rssMB‚ 2927.16796875# disk.nvme0c0n1.in‚ 6290.39453125 disk.dm-0.in‚ 6290.39453125 disk.sda3.out‚ 69.0390625 disk.dm-1.out‚ 69.0390625* proc.memory.percent‚17.866015434265137 cpu‚24.97204719092493 network.recv‚767084! disk.nvme0n1.in‚ 6290.39453125 disk.sda3.in‚0 disk.dm-1.in‚0†£‘¶‚(:† ñãèÔ—¹Ö­ gpu.0.gpu‚36.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.575' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.315 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á ïk·‚(:à ñãèÔÕ¢µ¯ disk.sda2.out‚0! disk.nvme0c0n1.in‚ 6435.765625 disk.nvme0n1.in‚ 6435.765625! disk.dm-0.out‚52748.509765625 disk.sda.out‚ 69.5703125( proc.memory.availableMB‚ 114.58203125 network.sent‚ 4.129204e+06 network.recv‚771788 disk.dm-1.out‚ 69.5703125 disk.sda1.in‚0$ memory_percent‚99.30064678192139 cpu‚24.972894608604335) disk./.usagePercent‚78.63115377598416 disk.sda2.in‚0& disk.nvme0c0n1.out‚52748.509765625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.3203125# proc.memory.rssMB‚ 2927.16796875* proc.memory.percent‚17.866015434265137 proc.cpu.threads‚17% disk./.usageGB‚1171.6610145568848$ disk.nvme0n1.out‚52748.509765625 disk.dm-0.in‚ 6435.765625 disk.sda3.in‚0 disk.sda3.out‚ 69.5703125 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.out‚0µL™½2¸- _wandbruntime‚2336  _runtime‚2336qPöǹÁ & lr-AdamW/pg1‚0.0007761700485946281 & lr-AdamW/pg2‚0.0007761700485946281 trainer/global_step‚2199 % _timestamp‚1.7884902365171185e+09 _runtime‚2336.157509778 _step‚175‚BÌºÆ % _timestamp‚1.7884902365171185e+09 _runtime‚2336.157509778 _step‚175 & lr-AdamW/pg1‚0.0007761700485946281 & lr-AdamW/pg2‚0.0007761700485946281 trainer/global_step‚2199¯Á;÷2»- _wandbruntime‚2336  _runtime‚2336ìOÒ–¼èO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151825472131584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48428555 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2240877548032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12073147 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171084180344320 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148029321842480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151825472131584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168744461992960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2339902008832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86377661 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2240104904192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48428555 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12073679 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151825490134528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33808436 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21734755 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2199 _step‚176 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151825490134528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37949108 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168744485933056 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154065577035776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21734757 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33807902 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2339718351360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37949108 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154066367682560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48451569 _runtime‚2336.158985218 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86400677 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2240877548032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12073147 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171084180344320 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148029321842480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151825472131584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168744461992960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2339902008832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86377661 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2240104904192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48428555 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12073679 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151825490134528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33808436 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21734755 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2199 _step‚176 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151825490134528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37949108 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168744485933056 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154065577035776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21734757 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33807902 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2339718351360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37949108 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154066367682560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516°©Ë:2¾- _wandbruntime‚2337  _runtime‚2337^h‰ø&¿òM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2241109649920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12078439 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151893172070912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151893172070912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33822842 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168819810335744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37965982 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220167168 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33823388 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196227072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86416246 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37965982 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48450266 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86439288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2241109649920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150329212352967 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 % _timestamp‚1.7884902374099393e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2241882307072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚148095388278240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151893190073856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2241882307072 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86439288 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12078983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151893190073856 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 _step‚177 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚154135072380928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21744405 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2340962175488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2337.050926326¥Ð´h À÷M G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86439288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2241109649920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150329212352967 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 % _timestamp‚1.7884902374099393e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2241882307072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚148095388278240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151893190073856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2241882307072 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86439288 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12078983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151893190073856 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 _step‚177 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚154135072380928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21744405 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2340962175488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2337.050926326 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2241109649920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12078439 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151893172070912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151893172070912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33822842 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168819810335744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37965982 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220167168 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33823388 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196227072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86416246 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37965982 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48450266 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489±™›ü2Á- _wandbruntime‚2337  _runtime‚23377KÐ@[ÂÕ 1loss/SRqqt2/detection_loss‚2.5387291703373194 -loss/FBt1/detection_loss‚2.47573401927948 % _timestamp‚1.7884902374123766e+09 /loss/FRt1/assignment_loss‚4.092456167274051 0loss/FRt2/assignment_loss‚2.9932838327744427 0loss/SRqqt1/detection_loss‚2.379750712164517 1loss/SRqqt3/detection_loss‚1.7677187034487725 trainer/global_step‚2199 epoch‚1 0loss/FRt4/assignment_loss‚1.3779456913471222 0loss/FBt1/assignment_loss‚0.6574509656056762 .loss/FRt1/detection_loss‚2.320974349313312 /loss/FRt2/detection_loss‚2.4666094341698814 /loss/FRt3/detection_loss‚2.2465415007256446 _runtime‚2337.052528325 2loss/SRqqt1/assignment_loss‚1.5595692953672902 1loss/FBt3/assignment_loss‚0.33785723231267184 &loss/total_loss‚42.888300026587636 2loss/SRqqt2/assignment_loss‚1.9332310215104371 0loss/FBt4/assignment_loss‚0.2342657661542281 1loss/SRqqt4/detection_loss‚0.8206135005210385 _step‚178 /loss/FBt3/detection_loss‚2.5007023334503176 /loss/FBt4/detection_loss‚0.6383832267851436 2loss/SRqqt4/assignment_loss‚1.0649248239091267 /loss/FRt4/detection_loss‚1.8683215941895137 .loss/FBt2/detection_loss‚2.875070595741272 /loss/FRt3/assignment_loss‚2.365870748428588 2loss/SRqqt3/assignment_loss‚1.2552046366780996 1loss/FBt2/assignment_loss‚0.11709070509969025Àóý¬`ÃÚ _runtime‚2337.052528325 2loss/SRqqt1/assignment_loss‚1.5595692953672902 1loss/FBt3/assignment_loss‚0.33785723231267184 &loss/total_loss‚42.888300026587636 2loss/SRqqt2/assignment_loss‚1.9332310215104371 0loss/FBt4/assignment_loss‚0.2342657661542281 1loss/SRqqt4/detection_loss‚0.8206135005210385 _step‚178 /loss/FBt3/detection_loss‚2.5007023334503176 /loss/FBt4/detection_loss‚0.6383832267851436 2loss/SRqqt4/assignment_loss‚1.0649248239091267 /loss/FRt4/detection_loss‚1.8683215941895137 .loss/FBt2/detection_loss‚2.875070595741272 /loss/FRt3/assignment_loss‚2.365870748428588 2loss/SRqqt3/assignment_loss‚1.2552046366780996 1loss/FBt2/assignment_loss‚0.11709070509969025 1loss/SRqqt2/detection_loss‚2.5387291703373194 -loss/FBt1/detection_loss‚2.47573401927948 % _timestamp‚1.7884902374123766e+09 /loss/FRt1/assignment_loss‚4.092456167274051 0loss/FRt2/assignment_loss‚2.9932838327744427 0loss/SRqqt1/detection_loss‚2.379750712164517 1loss/SRqqt3/detection_loss‚1.7677187034487725 trainer/global_step‚2199 epoch‚1 0loss/FRt4/assignment_loss‚1.3779456913471222 0loss/FBt1/assignment_loss‚0.6574509656056762 .loss/FRt1/detection_loss‚2.320974349313312 /loss/FRt2/detection_loss‚2.4666094341698814 /loss/FRt3/detection_loss‚2.2465415007256446²ÖHP£’Ä‚(:‡ €äèÔ˟Ю gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.237' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0474 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íÆ>jÅ‚(:ß €äèÔ°œÀ¯ disk.drbd1260.out‚ 0.3203125 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚99.30107593536377 cpu‚24.973680436155227 disk.dm-1.out‚ 69.9296875 network.sent‚ 4.135109e+06& disk.nvme0c0n1.out‚53212.513671875 disk.nvme0n1.in‚ 6435.765625 disk.dm-0.in‚ 6435.765625 disk.sda3.out‚ 69.9296875 disk.sda.in‚0 disk.sda1.in‚0% disk./.usageGB‚1171.7630729675293! disk.nvme0c0n1.in‚ 6435.765625 disk.sda1.out‚0( proc.memory.availableMB‚ 114.51171875 proc.cpu.threads‚17 disk.dm-1.in‚0 network.recv‚775761! disk.dm-0.out‚53212.513671875 disk.sda3.in‚0 disk.sda.out‚ 69.9296875# proc.memory.rssMB‚ 2927.16796875* proc.memory.percent‚17.866015434265137( disk./.usagePercent‚78.6380030015552$ disk.nvme0n1.out‚53212.513671875 disk.drbd1260.in‚0èp_’Æ‚(:‡ äèÔ‘¯§® gpu.0.gpu‚23.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.533' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7066 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éøP$jÇ‚(:ß äèÔ’Òƒ² cpu‚24.97461056700923 network.recv‚781340$ disk.nvme0n1.out‚53435.392578125! disk.dm-0.out‚53435.392578125 disk.drbd1260.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 4.242729e+06 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.32421875 disk.sda.in‚0 disk.sda.out‚ 70.3984375! disk.nvme0c0n1.in‚ 6435.765625$ memory_percent‚99.30109977722168) disk./.usagePercent‚78.65642299965134% disk./.usageGB‚1172.0375442504883 disk.dm-1.out‚ 70.3984375 disk.sda1.in‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚53435.392578125' proc.memory.availableMB‚ 114.5078125* proc.memory.percent‚17.866015434265137 disk.nvme0n1.in‚ 6435.765625 disk.dm-0.in‚ 6435.765625 disk.sda3.out‚ 70.3984375 disk.sda1.out‚0# proc.memory.rssMB‚ 2927.16796875ý:„È‚(:… žäèÔ«À ¯ gpu.0.gpu‚29.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.72' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úÌþAcÉ‚(:Ø žäèÔÄž¦³! proc.memory.availableMB‚114.5* proc.memory.percent‚17.866015434265137 proc.cpu.threads‚17$ disk.nvme0n1.out‚53640.927734375 disk.dm-0.in‚ 6435.765625 disk.sda3.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2927.16796875 cpu‚24.97535149599746 network.sent‚ 4.247784e+06 network.recv‚784870% disk./.usageGB‚1172.0365753173828& disk.nvme0c0n1.out‚53640.927734375! disk.dm-0.out‚53640.927734375 disk.sda3.out‚ 70.8359375 disk.dm-1.in‚0 disk.drbd1260.in‚0) disk./.usagePercent‚78.65635797373764! disk.nvme0c0n1.in‚ 6435.765625 disk.nvme0n1.in‚ 6435.765625 disk.drbd1260.out‚ 0.32421875 disk.sda.out‚ 70.8359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 70.8359375 disk.sda.in‚0# memory_percent‚99.3011474609375nYç=’Ê‚(:‡ ­äèÔž‡ù® gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚121.399' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2798 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÓåjË‚(:ß ­äèÔ™¶Ù²& disk.nvme0n1.out‚53984.95947265625# disk.dm-0.out‚53984.95947265625 cpu‚24.976122505546105 network.recv‚789508 disk.sda3.out‚ 71.203125 disk.dm-1.out‚ 71.203125 disk.drbd1260.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 114.49609375% disk./.usageGB‚1172.1664772033691 disk.sda3.in‚0 disk.drbd1260.out‚0.328125 disk.nvme0c0n1.in‚ 6532.65625( disk.nvme0c0n1.out‚53984.95947265625* proc.memory.percent‚17.866015434265137 network.sent‚ 4.253962e+06 disk.sda.out‚ 71.203125 disk.sda2.out‚0 disk.nvme0n1.in‚ 6532.65625 disk.dm-0.in‚ 6532.65625$ memory_percent‚99.30117130279541# proc.memory.rssMB‚ 2927.16796875 proc.cpu.threads‚17) disk./.usagePercent‚78.66507579830112 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0å­2Ì- _wandbruntime‚2385  _runtime‚2385ÊÖòÆÍÀ & lr-AdamW/pg1‚0.0007745181316775823 & lr-AdamW/pg2‚0.0007745181316775823 trainer/global_step‚2249 $ _timestamp‚1.788490285381009e+09 _runtime‚2385.021518329 _step‚179ÔÚ`!ËÎÅ trainer/global_step‚2249 $ _timestamp‚1.788490285381009e+09 _runtime‚2385.021518329 _step‚179 & lr-AdamW/pg1‚0.0007745181316775823 & lr-AdamW/pg2‚0.0007745181316775823³‘Ú–Q2Ï- _wandbruntime‚2385  _runtime‚2385ªë7í'ÐçO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49517457 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151332660669840 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157500825702400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196240384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38792806 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22217157 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 _runtime‚ 2385.02325958 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49540471 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚157500825702400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2392443202560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153615537150229 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2282876480389 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174904542515200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2282105831985 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚157501616349184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2291129248256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88310263 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220180480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151332643630480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49517457 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚155210469097984 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34558707 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 % _timestamp‚1.7884902853825917e+09üoÛw7ÑìO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚153614749462465 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2291129248256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88333279 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2392639442944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34558174 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚155210487100928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34558707 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 % _timestamp‚1.7884902853825917e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49517457 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151332660669840 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157500825702400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196240384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38792806 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22217157 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 _runtime‚ 2385.02325958 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49540471 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚157500825702400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2392443202560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153615537150229 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2282876480389 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174904542515200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2282105831985 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚157501616349184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2291129248256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88310263 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220180480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151332643630480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49517457 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚155210469097984 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88371890 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚174980944486400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚155278187040256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884902863494546e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚157570321047552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2393693071360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚2385.990596035 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12346859 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153682604726075 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151398727105600 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88348848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198324224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34573664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172587227474944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49562208 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚155278169037312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2283106971863 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12346307 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222264320 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49539168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22226803 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88348848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49539168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172587251415040 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34573110 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚157569530387456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2292134007296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49562208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34573110 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚157569530387456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2292134007296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49562208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88371890 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚174980944486400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚155278187040256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884902863494546e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚157570321047552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2393693071360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚2385.990596035 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12346859 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153682604726075 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151398727105600 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88348848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198324224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34573664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172587227474944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49562208 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚155278169037312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2283106971863 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12346307 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222264320 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49539168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22226803 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88348848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49539168µ•3­2Õ- _wandbruntime‚2385  _runtime‚2385OÅŠï^ÖØ 0loss/SRqqt4/detection_loss‚1.061204964845953 1loss/SRqqt2/assignment_loss‚2.184815776167494 epoch‚1 /loss/FBt1/detection_loss‚3.1463391333818436 /loss/FBt4/detection_loss‚0.6320951071592342 $ _timestamp‚1.788490286352638e+09 3loss/FBt3/assignment_loss‚0.0019637590785350767 1loss/SRqqt1/detection_loss‚2.4022909628622457 _runtime‚2385.992866574 2loss/SRqqt1/assignment_loss‚1.5308191252477241 1loss/SRqqt3/detection_loss‚1.5917187357297191 _step‚182 0loss/FRt3/assignment_loss‚2.5375832694548146 2loss/SRqqt4/assignment_loss‚1.0277006424911006 /loss/FBt1/assignment_loss‚0.111076644527202 /loss/FRt1/detection_loss‚2.5628814379806104 /loss/FBt3/detection_loss‚2.6619961261749268 0loss/FRt2/assignment_loss‚3.8076082413846795 .loss/FRt4/detection_loss‚1.379252676434558 trainer/global_step‚2249 2loss/SRqqt3/assignment_loss‚1.5148838179355317 /loss/FRt2/detection_loss‚2.5072161717848345 1loss/SRqqt2/detection_loss‚2.4429284009066494 /loss/FBt2/detection_loss‚3.0901534855365753 %loss/total_loss‚43.62484455961926 /loss/FRt1/assignment_loss‚3.577920216581096 0loss/FRt4/assignment_loss‚1.3086879253387451 1loss/FBt2/assignment_loss‚0.13140547429211438 1loss/FBt4/assignment_loss‚0.18918173409076933 /loss/FRt3/detection_loss‚2.2231207302323095z‰ 0c×Ý /loss/FRt1/assignment_loss‚3.577920216581096 0loss/FRt4/assignment_loss‚1.3086879253387451 1loss/FBt2/assignment_loss‚0.13140547429211438 1loss/FBt4/assignment_loss‚0.18918173409076933 /loss/FRt3/detection_loss‚2.2231207302323095 0loss/SRqqt4/detection_loss‚1.061204964845953 1loss/SRqqt2/assignment_loss‚2.184815776167494 epoch‚1 /loss/FBt1/detection_loss‚3.1463391333818436 /loss/FBt4/detection_loss‚0.6320951071592342 $ _timestamp‚1.788490286352638e+09 3loss/FBt3/assignment_loss‚0.0019637590785350767 1loss/SRqqt1/detection_loss‚2.4022909628622457 _runtime‚2385.992866574 2loss/SRqqt1/assignment_loss‚1.5308191252477241 1loss/SRqqt3/detection_loss‚1.5917187357297191 _step‚182 0loss/FRt3/assignment_loss‚2.5375832694548146 2loss/SRqqt4/assignment_loss‚1.0277006424911006 /loss/FBt1/assignment_loss‚0.111076644527202 /loss/FRt1/detection_loss‚2.5628814379806104 /loss/FBt3/detection_loss‚2.6619961261749268 0loss/FRt2/assignment_loss‚3.8076082413846795 .loss/FRt4/detection_loss‚1.379252676434558 trainer/global_step‚2249 2loss/SRqqt3/assignment_loss‚1.5148838179355317 /loss/FRt2/detection_loss‚2.5072161717848345 1loss/SRqqt2/detection_loss‚2.4429284009066494 /loss/FBt2/detection_loss‚3.0901534855365753 %loss/total_loss‚43.62484455961926¶k‰8gØ‚(:Ü ¼äèÔÑÕÀ¯ disk.sda3.out‚ 71.64453125 disk.dm-1.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.out‚ 54136.671875 disk.dm-0.in‚ 6554.90234375$ memory_percent‚99.29981231689453 cpu‚24.976837191939392 network.recv‚793859) disk./.usagePercent‚78.66513775212441% disk./.usageGB‚1172.1674003601074 disk.dm-1.out‚ 71.64453125 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.out‚ 54136.671875% proc.memory.availableMB‚ 114.71875 disk.sda.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 6554.90234375* proc.memory.percent‚17.866015434265137 network.sent‚ 4.259794e+06 disk.drbd1260.out‚0.328125 disk.sda.out‚ 71.64453125 disk.sda1.out‚0# disk.nvme0c0n1.out‚ 54136.671875! disk.nvme0n1.in‚ 6554.90234375# proc.memory.rssMB‚ 2927.16796875 proc.cpu.threads‚17 disk.sda3.in‚0™–žw’Ù‚(:‡ ¼äèÔ«Üî¯ gpu.0.gpu‚36.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.748' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bz‹’Ú‚(:‡ ËäèÔñç™® gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.623' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žþ*¡tÛ‚(:é ËäèÔ¶»†¯% disk.nvme0n1.out‚54411.1201171875 disk.sda3.out‚ 71.9921875 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2927.16796875* proc.memory.percent‚17.866015434265137" disk.dm-0.out‚54411.1201171875 disk.dm-1.out‚ 71.9921875 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 71.9921875( proc.memory.availableMB‚ 114.44921875 proc.cpu.threads‚17 network.recv‚800015% disk./.usageGB‚1172.0030364990234! disk.nvme0n1.in‚ 6554.90234375 disk.dm-0.in‚ 6554.90234375 disk.drbd1260.out‚ 0.33203125 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 6554.90234375' disk.nvme0c0n1.out‚54411.1201171875 network.sent‚ 4.367623e+06 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0$ memory_percent‚99.30145740509033 cpu‚24.977586893319906( disk./.usagePercent‚78.6541071554966ôF/a’Ü‚(:‡ ÚäèÔ鎂® gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.684' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÆûkÝ‚(:à ÚäèÔí¬³°# disk.dm-0.out‚54683.33154296875 disk.dm-1.out‚72.6875 disk.sda2.in‚0$ memory_percent‚99.30152893066406 network.recv‚803783& disk.nvme0n1.out‚54683.33154296875 disk.sda2.out‚0$ proc.memory.availableMB‚114.4375* proc.memory.percent‚17.866015434265137) disk./.usagePercent‚78.66922260834075% disk./.usageGB‚1172.2282676696777# disk.nvme0c0n1.in‚ 6554.90234375 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.33203125 disk.sda.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17! disk.nvme0n1.in‚ 6554.90234375 disk.sda3.in‚0 disk.sda3.out‚72.6875 disk.drbd1260.in‚0 disk.sda.out‚72.6875 disk.sda1.in‚0# proc.memory.rssMB‚ 2927.16796875 cpu‚24.978489825678054 network.sent‚ 4.372637e+06( disk.nvme0c0n1.out‚54683.33154296875 disk.dm-0.in‚ 6554.90234375dSÿ›2Þ- _wandbruntime‚2435  _runtime‚2435iÖFµÇßÁ & lr-AdamW/pg2‚0.0007727873573767297 trainer/global_step‚2299 % _timestamp‚1.7884903353724086e+09 _runtime‚2435.012786364 _step‚183 & lr-AdamW/pg1‚0.0007727873573767297"TÆÍÌàÆ & lr-AdamW/pg2‚0.0007727873573767297 trainer/global_step‚2299 % _timestamp‚1.7884903353724086e+09 _runtime‚2435.012786364 _step‚183 & lr-AdamW/pg1‚0.0007727873573767297·GÃR2á- _wandbruntime‚2435  _runtime‚2435Ãn<º âèO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154635982457840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2340608304640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50606359 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50629373 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 _step‚184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90242865 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2341380948480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39636508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176279296271360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚2435.014156626 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158595484067328 ZFDeviceStatsMonitor.on_train_batch_stáèÀ|4art/inactive_split_bytes.all.freed‚178724477667328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2332947065497 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160936074369024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158595484067328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160936074369024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50606359 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160936865015808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12609401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154635965418480 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35308435 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 % _timestamp‚1.7884903353736563e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12608880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22699555 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚156968141835573 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2445367150592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156968929523337 trainer/global_step‚2299 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39636506 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39636508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158595466064384 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 % _timestamp‚1.7884903353736563e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12608880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22699555 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚156968141835573 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2445367150592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156968929523337 trainer/global_step‚2299 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39636506 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39636508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158595466064384 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2340608304640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50606359 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50629373 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 _step‚184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90242865 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2341380948480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39636508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176279296271360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚2435.014156626 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158595484067328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚178724477667328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2332947065497 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160936074369024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158595484067328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160936074369024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50606359 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160936865015808¸Ûh2ä- _wandbruntime‚2435  _runtime‚2435é‘•¢ø&åòM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2435.906116417 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2341613050368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22709203 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183644160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50651110 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2341613050368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 % _timestamp‚1.7884903362652886e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚178800884452352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39653382 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39653380 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35323410 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚178801092036608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161005569714176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22709205 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12614718 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161005569714176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2342385707520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50628070 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176354668554240 _step‚185 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2446423482368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2333948205583 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157035209411211 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158663166003712 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90304492 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39653380 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2446239838208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚154702048893600 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158663166003712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176354644614144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005ZFÄ’Œæ÷M XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161005569714176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22709205 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12614718 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161005569714176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2342385707520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50628070 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176354668554240 _step‚185 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2446423482368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2333948205583 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157035209411211 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15866316600371Ø:̶q2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90304492 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39653380 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2446239838208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚154702048893600 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158663166003712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176354644614144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2435.906116417 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2341613050368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22709203 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183644160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50651110 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2341613050368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 % _timestamp‚1.7884903362652886e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚178800884452352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39653382 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39653380 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35323410 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚178801092036608¹¯sñÇ2ç- _wandbruntime‚2435  _runtime‚2435-–ZèÔ 2loss/SRqqt1/assignment_loss‚1.6247538049416295 1loss/FBt3/assignment_loss‚0.10807302046386111 trainer/global_step‚2299 0loss/FRt4/assignment_loss‚1.1967694484270537 0loss/SRqqt3/detection_loss‚1.715212731820638 .loss/FBt1/detection_loss‚2.765181907585689 _runtime‚2435.907811111 0loss/SRqqt2/assignment_loss‚2.00050526156145 1loss/SRqqt4/detection_loss‚1.2712524096289324 _step‚186 0loss/FRt1/assignment_loss‚3.2555790742238364 2loss/SRqqt3/assignment_loss‚1.8213994035222492 0loss/SRqqt1/detection_loss‚2.238257764742292 0loss/SRqqt2/detection_loss‚2.343816159402623 epoch‚1 0loss/FRt2/assignment_loss‚3.2710331960728296 0loss/FRt3/assignment_loss‚2.8345879676730132 .loss/FRt3/detection_loss‚2.510274652478307 /loss/FBt4/detection_loss‚0.6004898173318423 %loss/total_loss‚43.99160875108717 1loss/FBt1/assignment_loss‚0.32351724790143116 .loss/FRt1/detection_loss‚2.643007720096244 /loss/FBt2/detection_loss‚2.8752718908446178 % _timestamp‚1.7884903362676528e+09 2loss/SRqqt4/assignment_loss‚1.0173134432129918 1loss/FBt2/assignment_loss‚0.31341228175110053 0loss/FBt4/assignment_loss‚0.2815316080735115 /loss/FBt3/detection_loss‚2.5370144844055176 /loss/FRt2/detection_loss‚2.5147625239271867 /loss/FRt4/detection_loss‚1.9285909309983253EA_éÙ _runtime‚2435.907811111 0loss/SRqqt2/assignment_loss‚2.00050526156145 1loss/SRqqt4/detection_loss‚1.2712524096289324 _step‚186 0loss/FRt1/assignment_loss‚3.2555790742238364 2loss/SRqqt3/assignment_loss‚1.8213994035222492 0loss/SRqqt1/detection_loss‚2.238257764742292 0loss/SRqqt2/detection_loss‚2.343816159402623 epoch‚1 0loss/FRt2/assignment_loss‚3.2710331960728296 0loss/FRt3/assignment_loss‚2.8345879676730132 .loss/FRt3/detection_loss‚2.510274652478307 /loss/FBt4/detection_loss‚0.6004898173318423 %loss/total_loss‚43.99160875108717 1loss/FBt1/assignment_loss‚0.32351724790143116 .loss/FRt1/detection_loss‚2.643007720096244 /loss/FBt2/detection_loss‚2.8752718908446178 % _timestamp‚1.7884903362676528e+09 2loss/SRqqt4/assignment_loss‚1.0173134432129918 1loss/FBt2/assignment_loss‚0.31341228175110053 0loss/FBt4/assignment_loss‚0.2815316080735115 /loss/FBt3/detection_loss‚2.5370144844055176 /loss/FRt2/detection_loss‚2.5147625239271867 /loss/FRt4/detection_loss‚1.9285909309983253 2loss/SRqqt1/assignment_loss‚1.6247538049416295 1loss/FBt3/assignment_loss‚0.10807302046386111 trainer/global_step‚2299 0loss/FRt4/assignment_loss‚1.1967694484270537 0loss/SRqqt3/detection_loss‚1.715212731820638 .loss/FBt1/detection_loss‚2.765181907585689ºµŽNÊ’ê‚(:‡ éäèÔÇ‚¯­ gpu.0.gpu‚16.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å7Zlë‚(:á éäèÔ爦± disk.drbd1260.out‚ 0.3359375$ memory_percent‚99.30191040039062% disk./.usageGB‚1172.4068489074707 disk.sda2.in‚0 disk.dm-0.in‚ 6554.90234375 disk.dm-1.in‚0 disk.dm-1.out‚ 73.078125* proc.memory.percent‚17.866015434265137 cpu‚24.979193224091198 network.sent‚ 4.378963e+06 disk.sda2.out‚0% disk.nvme0n1.out‚55257.5087890625" disk.dm-0.out‚55257.5087890625 disk.sda.in‚0 disk.sda.out‚ 73.078125# proc.memory.availableMB‚114.375 network.recv‚808454 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 6554.90234375 disk.sda3.out‚ 73.078125# proc.memory.rssMB‚ 2927.16796875 proc.cpu.threads‚17) disk./.usagePercent‚78.68120734504868 disk.sda1.out‚0' disk.nvme0c0n1.out‚55257.5087890625! disk.nvme0n1.in‚ 6554.90234375 disk.sda3.in‚0 disk.drbd1260.in‚0ÀÎ#-’ì‚(:‡ øäèԫ® gpu.0.gpu‚38.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.472' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.8944 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í"Imí‚(:â øäèԢͩ² network.recv‚814748) disk./.usagePercent‚78.64559132087763 disk.sda3.out‚ 73.484375 disk.drbd1260.out‚ 0.3359375 disk.sda.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 6682.1328125 disk.nvme0n1.in‚ 6682.1328125 network.sent‚ 4.48717e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 73.484375 disk.sda1.in‚0' disk.nvme0c0n1.out‚55655.4501953125% disk.nvme0n1.out‚55655.4501953125$ memory_percent‚99.30346012115479# proc.memory.rssMB‚ 2927.16796875% disk./.usageGB‚1171.8761444091797 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 6682.1328125( proc.memory.availableMB‚ 114.12109375 cpu‚24.979885535046936 proc.cpu.threads‚17 disk.sda.out‚ 73.484375 disk.sda2.out‚0" disk.dm-0.out‚55655.4501953125* proc.memory.percent‚17.866015434265137Ø@?·œî‚(:‘ ‡åèÔÌ“»­ gpu.0.gpu‚15.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.831' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚32.96619999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿|¤”oï‚(:ä ‡åèÔÝÛæ¯" disk.nvme0c0n1.in‚ 6682.1328125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 73.8359375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.33984375 disk.sda1.out‚0 cpu‚24.980543790562965 network.recv‚818543 disk.nvme0n1.in‚ 6682.1328125 disk.dm-0.in‚ 6682.1328125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.866015434265137 network.sent‚ 4.492353e+06) disk./.usagePercent‚78.65152071137817 disk.sda3.out‚ 73.8359375 disk.sda1.in‚0$ memory_percent‚99.30193424224854( proc.memory.availableMB‚ 114.37109375% disk./.usageGB‚1171.9644966125488& disk.nvme0c0n1.out‚56003.095703125$ disk.nvme0n1.out‚56003.095703125! disk.dm-0.out‚56003.095703125 disk.sda.out‚ 73.8359375# proc.memory.rssMB‚ 2927.16796875 proc.cpu.threads‚17·M¼"2ð- _wandbruntime‚2489  _runtime‚2489üÛ¯*ÇñÁ & lr-AdamW/pg1‚0.0007709780863828406 & lr-AdamW/pg2‚0.0007709780863828406 trainer/global_step‚2349 % _timestamp‚1.7884903894946375e+09 _runtime‚2489.135164462 _step‚187*s¢ÌòÆ & lr-AdamW/pg2‚0.0007709780863828406 trainer/global_step‚2349 % _timestamp‚1.7884903894946375e+09 _runtime‚2489.135164462 _step‚187 & lr-AdamW/pg1‚0.0007709780863828406»Ã:%h2ó- _wandbruntime‚2489  _runtime‚2489zÀ1î'ôèO IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164371323035648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2497914151424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180046737350656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160321534208681 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 trainer/global_step‚2349 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2390860004864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23181957 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157939304245840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164371323035648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚161980463030784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164372113682432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51695261 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40480206 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161980481033728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161980463030784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51718275 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160322321896445 % _timestamp‚1.7884903894960387e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2390860004864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12876162 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2498099906048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182544837256704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40480208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36058117 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157939287206480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2391632648704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23181955 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185754624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2391632648704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164372113682432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36058684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12876727 _runtime‚2489.136652166 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51695261 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚567 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92175467 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _step‚188 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209694720 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚565 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180046713410560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51718275 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92198483 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164371323035648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2497914151424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180046737350656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160321534208681 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 trainer/global_step‚2349 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2390860004864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23181957 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157939304245840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164371323035648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚161980463030784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164372113682432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51695261 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40480206 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161980481033728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161980463030784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51718275 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160322321896445 % _timestamp‚1.7884903894960387e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2390860004864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12876162 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2498099906048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182544837256704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40480208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36058117 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157939287206480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2391632648704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23181955 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185754624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2391632648704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164372113682432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36058684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12876727 _runtime‚2489.136652166 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51695261 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚567 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92175467 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _step‚188 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209694720 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚565 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180046713410560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51718275 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772643840 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92198483 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352¼¶äªpMö‚(:‡ –åèÔÉé»° gpu.0.gpu‚40.0 gpu.0.memory‚5+ gpu.0.meµÞEmoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.909' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.5818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬™Ó!i÷‚(:Þ –åèÔî³²" disk.nvme0c0n1.in‚ 6682.1328125! disk.dm-0.out‚56100.529296875 disk.sda3.out‚ 74.37890625 disk.sda.out‚ 74.37890625 disk.sda2.in‚0 cpu‚24.9812857976684 disk.nvme0n1.in‚ 6682.1328125 disk.dm-0.in‚ 6682.1328125 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.866015434265137 proc.cpu.threads‚17 network.recv‚822148' disk./.usagePercent‚78.685078946989& disk.nvme0c0n1.out‚56100.529296875 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.33984375 disk.sda2.out‚0$ memory_percent‚99.30496215820312% disk./.usageGB‚1172.4645385742188$ disk.nvme0n1.out‚56100.529296875 disk.sda3.in‚0 disk.dm-1.out‚ 74.37890625 disk.sda1.in‚0# proc.memory.availableMB‚113.875# proc.memory.rssMB‚ 2927.16796875 network.sent‚ 4.510451e+06&Çú„2ø- _wandbruntime‚2490  _runtime‚2490ì81Vø&ùòM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12882110 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92237094 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2391864750592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40497080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36073715 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36073163 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180122085693440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2490.217648738 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2392637407744 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2499160179712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160388601784319 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158005353642240 trainer/global_step‚2349 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164440818380800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162048180973056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40497080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2391864750592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92237094 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51740012 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2383248142079 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23191605 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40497082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12881560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162048180973056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2392637407744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2ìrv)ý&ú÷M TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222264320 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _step‚189 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23191603 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180122061753344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51716972 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51716972 % _timestamp‚1.7884903905763292e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92214052 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40497082 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162048162970112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019673600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164440027720704 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164440818380800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2384018790691 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚182621023608832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36073163 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180122085693440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2490.217648738 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2392637407744 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2499160179712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160388601784319 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158005353642240 trainer/global_step‚2349 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164440818380800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162048180973056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687972 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40497080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2391864750592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92237094 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51740012 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2383248142079 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23191605 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40497082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12881560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162048180973056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2392637407744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12882110 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634817 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92237094 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2391864750592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40497080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36073715 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167806571702272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 L¸®« /DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _step‚192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52807177 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186364871075840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165365459997184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2432317587309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41323908 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚167806571702272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52784163 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2550930475008 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165365478000128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2441111705088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚564 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 2538.06501082 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41323906 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13144375 % _timestamp‚1.7884904384240994e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167807362349056 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165365478000128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41323908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23664355 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94108069 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13144939 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚183814154489856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2433088235713_M:¯ò'‰ìO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 2538.06501082 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41323906 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13144375 % _timestamp‚1.7884904384240994e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167807362349056 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165365478000128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41323908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23664355 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94108069 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13144939 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772643840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882817 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚183814154489856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2433088235713 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2441111705088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚161242608994480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41323906 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2441884348928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165365459997184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790646784 trainer/global_step‚2399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23664357 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687764 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚161242626033840 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167806571702272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _step‚192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52807177 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019673600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186364871075840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165365459997184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2432317587309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41323908 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23014 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648404 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚167806571702272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28113 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52784163 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2550930475008 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28113 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634817 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165365478000128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2441111705088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚564 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790646784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23014 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312ÀJÑ­2Š- _wandbruntime‚2539  _runtime‚2539›ˆªø&‹òM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52828914 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165433159936512 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94169696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41340782 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13149759 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167876067047424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52828914 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41340780 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36824311 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163742781845399 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2433318727187 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94146654 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36823762 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23674003 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚183889478892544 _runtime‚2539.047430362 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186441284026368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13150306 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚183889502832640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2551995069440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2551805133824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2434089375799 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41340782 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167876067047424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2442116450816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚161308692469600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2442889107968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163741994157427 trainer/global_step‚2399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213875712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189935616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161308675430240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52805874 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23674005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚165433177939456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚165433177939456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186441497902080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94146654 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94169696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52805874 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005…V<Ï’Œ÷M WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165433159936512 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94169696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41340782 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13149759 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167876067047424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52828914 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23042 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41340780 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36824311 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163742781845399 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2433318727187 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94146654 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36823762 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23674003 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚183889478892544 _runtime‚2539.047430362 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186441284026368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13150306 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚183889502832640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2551995069440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2551805133824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2434089375799 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41340782 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167876067047424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2442116450816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚161308692469600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2442889107968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882817 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28113 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163741994157427 trainer/global_step‚2399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28113 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213875712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189935616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161308675430240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52805874 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23674005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648612 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚165433177939456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚165433177939456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186441497902080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94146654 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94169696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52805874 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52828914Á±n·2- _wandbruntime‚2539  _runtime‚2539]i ÏWŽÑ /loss/FRt2/detection_loss‚2.3989598859440195 0loss/SRqqt2/detection_loss‚2.446314760867287 1loss/SRqqt3/detection_loss‚1.6393187974507992 %loss/total_loss‚43.65144194580262 % _timestamp‚1.7884904394085405e+09 /loss/FRt3/assignment_loss‚2.316298408167703 /loss/FRt4/assignment_loss‚1.112849049270153 0loss/FBt1/assignment_loss‚0.4760011501610279 /loss/FBt2/assignment_loss‚0.354541209298821 /loss/FRt3/detection_loss‚2.4300051856608618 1loss/SRqqt4/detection_loss‚0.8830389131766718 .loss/FRt2/assignment_loss‚3.21363617254026 1loss/SRqqt3/assignment_loss‚1.528151139165633 2loss/SRqqt1/assignment_loss‚1.8821815248046603 1loss/SRqqt1/detection_loss‚2.4514420458248685 /loss/FBt4/detection_loss‚0.6862848031465192 epoch‚1 _runtime‚2539.049039464 _step‚194 1loss/FBt4/assignment_loss‚0.17930610861401877 2loss/SRqqt4/assignment_loss‚0.9718804410407068 0loss/FRt1/assignment_loss‚3.9806731287389994 /loss/FRt1/detection_loss‚2.8539454508572817 .loss/FRt4/detection_loss‚1.733462800970301 .loss/FBt2/detection_loss‚2.381220904263583 trainer/global_step‚2399 0loss/FBt3/assignment_loss‚0.2870097506905343 /loss/FBt1/detection_loss‚2.8355658769607546 /loss/FBt3/detection_loss‚2.7255750795205436 2loss/SRqqt2/assignment_loss‚1.8837793586666094•v̼\Ö 2loss/SRqqt1/assignment_loss‚1.8821815248046603 1loss/SRqqt1/detection_loss‚2.4514420458248685 /loss/FBt4/detection_loss‚0.6862848031465192 epoch‚1 _runtime‚2539.049039464 _step‚194 1loss/FBt4/assignment_loss‚0.17930610861401877 2loss/SRqqt4/assignment_loss‚0.9718804410407068 0loss/FRt1/assignment_loss‚3.9806731287389994 /loss/FRt1/detection_loss‚2.8539454508572817 .loss/FRt4/detection_loss‚1.733462800970301 .loss/FBt2/detection_loss‚2.381220904263583 trainer/global_step‚2399 0loss/FBt3/assignment_loss‚0.2870097506905343 /loss/FBt1/detection_loss‚2.8355658769607546 /loss/FBt3/detection_loss‚2.7255750795205436 2loss/SRqqt2/assignment_loss‚1.8837793586666094 /loss/FRt2/detection_loss‚2.3989598859440195 0loss/SRqqt2/detection_loss‚2.446314760867287 1loss/SRqqt3/detection_loss‚1.6393187974507992 %loss/total_loss‚43.65144194580262 % _timestamp‚1.7884904394085405e+09 /loss/FRt3/assignment_loss‚2.316298408167703 /loss/FRt4/assignment_loss‚1.112849049270153 0loss/FBt1/assignment_loss‚0.4760011501610279 /loss/FBt2/assignment_loss‚0.354541209298821 /loss/FRt3/detection_loss‚2.4300051856608618 1loss/SRqqt4/detection_loss‚0.8830389131766718 .loss/FRt2/assignment_loss‚3.21363617254026 1loss/SRqqt3/assignment_loss‚1.528151139165633Â?ùPœ‚(:‘ ÒåèÔ²ͮ gpu.0.gpu‚40.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚150.32' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.063999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ãúþt‘‚(:é ÒåèÔ“Øᯠdisk.drbd1260.out‚ 0.34765625 disk.sda1.in‚0! disk.nvme0n1.in‚ 6805.15234375% disk.nvme0n1.out‚57156.8037109375 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 6805.15234375" disk.dm-0.out‚57156.8037109375$ memory_percent‚99.30262565612793# proc.memory.rssMB‚ 2927.16796875) disk./.usagePercent‚78.69843639609552# disk./.usageGB‚1172.66357421875 disk.sda3.in‚0 disk.sda3.out‚ 76.12109375 disk.drbd1260.in‚0 disk.sda.out‚ 76.12109375 disk.sda2.out‚0' disk.nvme0c0n1.out‚57156.8037109375 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 6805.15234375' proc.memory.availableMB‚ 114.2578125* proc.memory.percent‚17.866015434265137 cpu‚24.98404628999671 proc.cpu.threads‚17 network.sent‚ 4.701602e+06 network.recv‚840264 disk.dm-1.in‚0 disk.dm-1.out‚ 76.12109375­Kƒœ’‚(:‘ áåèÔÓ¬“° gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.09' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.418000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0síRwr“‚(:ç áåèÔËÚš´# memory_percent‚99.3100643157959* proc.memory.percent‚17.874908447265625 cpu‚25.000872245797773 network.recv‚846692 disk.dm-1.in‚0 disk.dm-1.out‚ 76.77734375 disk.sda1.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚2928.625 proc.cpu.threads‚17) disk./.usagePercent‚78.70701316050763$ disk./.usageGB‚1172.791374206543 disk.drbd1260.in‚0 disk.sda.in‚0( disk.nvme0c0n1.out‚57529.60302734375& disk.nvme0n1.out‚57529.60302734375 disk.dm-0.in‚ 6805.15234375' proc.memory.availableMB‚ 113.0390625 network.sent‚ 4.810253e+06 disk.sda3.in‚0 disk.sda3.out‚ 76.77734375 disk.drbd1260.out‚ 0.3515625 disk.sda.out‚ 76.77734375 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 6805.15234375! disk.nvme0n1.in‚ 6805.15234375# disk.dm-0.out‚57529.60302734375‚—=Á‘”‚(:† ðåèÔ©¸Õ® gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚176.264' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ewês•‚(:è ðåèÔ«¿ý²# disk.dm-0.out‚57726.67724609375' proc.memory.availableMB‚ 112.7734375 proc.cpu.threads‚17 network.sent‚ 4.816113e+06 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3515625 disk.sda.out‚ 77.20703125 disk.sda1.in‚0 proc.memory.rssMB‚2928.625* proc.memory.percent‚17.874908447265625 network.recv‚851242) disk./.usagePercent‚78.70728043237337 disk.dm-1.out‚ 77.20703125 disk.sda2.out‚0& disk.nvme0n1.out‚57726.67724609375$ memory_percent‚99.31168556213379 cpu‚25.03040017446032% disk./.usageGB‚1172.7953567504883 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 6951.06640625( disk.nvme0c0n1.out‚57726.67724609375 disk.sda3.out‚ 77.20703125 disk.sda2.in‚0! disk.nvme0n1.in‚ 6951.06640625 disk.dm-0.in‚ 6951.06640625HH,z’–‚(:‡ ÿåèÔ¯»ø® gpu.0.gpu‚45.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.281' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0562 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FüöÈs—‚(:è ÿåèÔü‹á² disk.sda1.in‚0 proc.memory.rssMB‚2928.625 proc.cpu.threads‚17 network.sent‚ 4.821252e+06) disk./.usagePercent‚78.72405430202434# disk.nvme0c0n1.in‚ 6951.06640625! disk.nvme0n1.in‚ 6951.06640625 disk.sda3.out‚ 77.55078125 disk.sda.out‚ 77.55078125 disk.sda1.out‚0 disk.sda2.out‚0 network.recv‚854804# disk.dm-0.out‚58039.64013671875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0% proc.memory.availableMB‚ 112.78125 cpu‚25.062847733600304 disk.sda3.in‚0 disk.dm-1.out‚ 77.55078125$ memory_percent‚99.31163787841797* proc.memory.percent‚17.874908447265625% disk./.usageGB‚1173.0452995300293( disk.nvme0c0n1.out‚58039.64013671875& disk.nvme0n1.out‚58039.64013671875 disk.dm-0.in‚ 6951.06640625 disk.drbd1260.out‚ 0.35546875<¸“r‘˜‚(:† ŽæèÔ³Š˜® gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚169.981' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9962 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹©©ïu™‚(:ê ŽæèÔÒ¿»¯ disk.sda.out‚ 78.25390625 disk.sda1.in‚0( proc.memory.availableMB‚ 112.52734375 proc.cpu.threads‚17 network.recv‚858290( disk./.usagePercent‚78.7273946883311! disk.nvme0n1.in‚ 6951.06640625 disk.dm-1.out‚ 78.25390625 disk.drbd1260.out‚ 0.35546875 proc.memory.rssMB‚2928.625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0$ memory_percent‚99.31318759918213* proc.memory.percent‚17.874908447265625 cpu‚25.092669975955157 network.sent‚ 4.828112e+06% disk./.usageGB‚1173.0950736999512( disk.nvme0c0n1.out‚58478.47216796875 disk.sda3.out‚ 78.25390625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 6951.06640625& disk.nvme0n1.out‚58478.47216796875 disk.dm-0.in‚ 6951.06640625# disk.dm-0.out‚58478.47216796875 ÷2š- _wandbruntime‚2615  _runtime‚2615¯0µU*›ÏT ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 580043264 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚95906104 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578072496 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 11892003328 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1005 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1005 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚42280140 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3125 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚53607749 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2483184355328 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚13495611 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 trainer/global_step‚2417 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27844 _runtime‚2615.572947703 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚24238687 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚171363473208320 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚42280140 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27844 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2483184355328 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18213 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2483764398592 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 580043264 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 371748352 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚53625962 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 10929229312 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚168879690806784 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚53607749 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882817 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 12155092992 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚42280142 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚24238689 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚190141723918848 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚168879708809728 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2594021797376 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚95906104 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚37734120 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚187547678181376 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18215 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 146442752 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚190141577476096 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12155092992 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2483764398592 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 598046208 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚516 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚171362875162112 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 11892003328 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28113 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚13495433 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚167044612603539 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18215 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚168879708809728 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚180 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚187547702121472 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111856 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚164570384477984 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2474823237411 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 % _timestamp‚1.7884905159316034e+09 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 598046208 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚171362875162112 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634817 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1005%Zy…#œÔT MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18215 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚168879708809728 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚180 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚187547702121472 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111856 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚164570384477984 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2474823237411 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 % _timestamp‚1.7884905159316034e+09 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 598046208 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚171362875162112 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634817 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1005 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 580043264 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚95906104 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578072496 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 11892003328 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1005 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1005 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚42280140 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3125 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚53607749 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2483184355328 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚13495611 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 trainer/global_step‚2417 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27844 _runtime‚2615.572947703 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚24238687 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚171363473208320 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019673600 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚42280140 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27844 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2483184355328 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18213 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2483764398592 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 580043264 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 371748352 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚53625962 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 10929229312 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚168879690806784 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚53607749 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882817 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 12155092992 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚42280142 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚24238689 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚190141723918848 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚168879708809728 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2594021797376 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚95906104 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚37734120 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚187547678181376 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18215 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_V$¼ÝÕbytes.all.current‚ 146442752 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚190141577476096 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12155092992 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2483764398592 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 598046208 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚516 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚171362875162112 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 11892003328 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28113 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚13495433 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚167044612603539ÚÑ=.2- _wandbruntime‚2616  _runtime‚2616m’¹Ó^)žØR D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28113 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28113 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18213 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚39 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚164612663278720 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1005 _runtime‚2616.934244007 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580043264 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚164612680318080 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 598046208 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚95927370 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚42293112 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 598046208 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚168925176808960 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595111856 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚187594680266752 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚13500698 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚167087966716455 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3125 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 580043264 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2594486545920 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚171408807909888 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1019673600 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚171408807909888 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚180 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3125 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚42293114 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚53616043 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚167087371604599 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1005 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚516 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚53634256 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚168925158806016 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1075 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2594364043264 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 578072496 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892003328 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚178 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚171409405956096 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2484229147136 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚168925176808960 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 $ _timestamp‚1.788490517293041e+09 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1005 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2417 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚95909155 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚168925158806016 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 122502656 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚53616043 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11129585664 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2483649103872 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚190189166812672 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2474708325879 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2484229147136 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚42293114 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 11518882817 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2475286398375 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚171409405956096 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18215 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚37747365 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚24246669 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚190189020369920 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 371748352 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚187594656326656 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚95927370 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 11892003328 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚24246667 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18213 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚37747545 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚42293112 _step‚196 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27844 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚187594680266752 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚13500698 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚167087966716455 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3125 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 580043264 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2594486545920 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚171408807909888 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1019673600 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚171408807909888 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚180 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3125 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚42293114 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚53616043 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚167087371604599 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1005 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚516 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚53634256 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚168925158806016 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1075 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2594364043264 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 578072496 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892003328 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚178 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚171409405956096 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2484229147136 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚168925176808960 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019673600 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 $ _timestamp‚1.788490517293041e+09 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1005 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2417 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚95909155 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚168925158806016 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 122502656 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚53616043 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11129585664 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2483649103872 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚190189166812672 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2474708325879 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2484229147136 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚42293114 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 11518882817 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2475286398375 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚171409405956096 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18215 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚37747365 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚24246669 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚190189020369920 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 371748352 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚187594656326656 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚95927370 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 11892003328 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚24246667 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18213 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚37747545 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚42293112 _step‚196 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27844 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28113 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18213 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚39 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚164612663278720Ä!êQŸ ‚(:’ æèÔãØž¯ gpu.0.gpu‚49.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚174.592' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.918400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`J)}g¡‚(:Ü æèÔˆÕà± cpu‚25.12186887776475 disk.sda1.in‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 112.2421875* proc.memory.percent‚17.874908447265625 network.recv‚862913 disk.sda.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 6951.06640625 disk.dm-0.in‚ 6951.06640625 disk.drbd1260.out‚ 0.35546875 proc.memory.rssMB‚2928.625 proc.cpu.threads‚17 network.sent‚ 4.845499e+06) disk./.usagePercent‚78.73929161445479 disk.sda.out‚ 78.6484375 disk.sda1.out‚0 disk.sda2.in‚0% disk.nvme0c0n1.out‚59085.98828125 disk.sda3.out‚ 78.6484375 disk.dm-1.out‚ 78.6484375$ disk./.usageGB‚1173.272346496582! disk.nvme0n1.in‚ 6951.06640625# disk.nvme0n1.out‚59085.98828125 disk.dm-0.out‚59085.98828125 disk.sda3.in‚0 disk.drbd1260.in‚0$ memory_percent‚99.31492805480957Ssx2¢- _wandbruntime‚2632  _runtime‚2632TV4öÊ £Ž¦ ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15777777135372162 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.7777777910232544 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.019592054188251495 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9791848659515381 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1980479210615158 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.36732301115989685 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18330642580986023 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5969685316085815 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.44342759251594543 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3336106538772583 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.33981481194496155 epoch‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 _runtime‚2632.948358242 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1520296037197113 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22268253564834595 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.389428973197937 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.44272997975349426 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4486314356327057 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9596161842346191 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5001489520072937 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.31494253873825073 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9871404767036438 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.378361314535141 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4099999964237213 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.42572179436683655 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19703224301338196 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.42635443806648254 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24945589900016785 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4866733253002167 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.25925925374031067 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.10000000149011612 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.34651413559913635 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.21307794749736786 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_6‚1 particle/accuracy_7_of_7‚0.5 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚43.33010752164224 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4422617256641388 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8148845434188843 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5535079836845398 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0921969786286354 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_5_of_5‚1 jet/accuracy_4_of_6‚0 particle/accuracy_4_of_7‚1 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3215548098087311 particle/accuracy_2_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3977954387664795 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015004695393145084 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936455488204956 ,particle/sensitivity‚0.25428101420402527 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.997300386428833 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4360472857952118 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_4_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285714030265808 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99541836977005 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3342774510383606 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10831031948328018 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 +particle/specificity‚0.9524706602096558 9"Purity/*FRt4ëNECKrSRqqt0FBt/SRqqt_purity‚0.2777777910232544 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9876121878623962 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.41396522521972656 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2976190447807312 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.39578917622566223 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.45133933424949646 4 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3928242623806 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04293462634086609 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9920635223388672 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_2_of_4‚0.3653026819229126 *jet/accuracy_3_of_5‚0.0964093804359436 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 jet/accuracy_6_of_6‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.972787082195282 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.40442955493927 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.37878790497779846 particle/accuracy_1_of_3‚1 (particle/accuracy‚0.8434503078460693 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29660114645957947 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4228268563747406 +jet/accuracy_3_of_6‚0.03846153989434242 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 _step‚197 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4696170687675476 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.2074723094701767 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9857617020606995 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996565580368042 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23900601267814636 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011022928170859814 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07183908671140671 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5219634175300598 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7651515603065491 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.20133531093597412 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.022641509771347046 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.46162280440330505 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10451014339923859 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04047500342130661 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.41034483909606934 particle/accuracy_4_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17281650006771088 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10201013833284378 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3536613881587982 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09374584257602692 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333730697632 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9748062491416931 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9887301921844482 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.20588311553001404 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.31651654839515686 +jet/accuracy_2_of_6‚0.46794870495796204 jet/accuracy_1_of_7‚1 particle/accuracy_2_of_7‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24871937930583954 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.483682245016098 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.17688968777656555 +jet/accuracy_3_of_4‚0.06063299998641014 *jet/accuracy_2_of_5‚0.4810164272785187 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39444446563720703 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4103710651397705 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9868301749229431 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20567075908184052 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9150793552398682 particle/accuracy_3_of_7‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03213656321167946 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9837325215339661 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3480391800403595 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9857972860336304 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.400726854801178 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2122131884098053 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2992468774318695 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03896718472242355 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9916955828666687 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4294070303440094 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3143274784088135 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 trainer/global_step‚2417 % _timestamp‚1.7884905333054633e+09 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23457522690296173 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3166666626930237 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1763686239719391 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43798449635505676 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4454023241996765 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833134651184 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3932715654373169 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9956604838371277 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5975695252418518 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2777777910232544 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4864003360271454 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9510416984558105 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_6‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_6_of_7‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37931543588638306 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.39531755447387695 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5705586671829224 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3220538794994354 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.41833844780921936 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5523361563682556 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.38633492588996887 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.39935997128486633 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.984059751033783 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4652778208255768 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20567075908184052 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0384608618915081 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757660865784 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2990013659000397 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02371695265173912 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.995908260345459 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.016599705442786217 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4368303716182709 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3828769624233246 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08040229976177216 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 jet/accuracy_5_of_5‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.416631281375885 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9285714030265808 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04863031208515167 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09047619253396988 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9748587012290955 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9748773574829102 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.40189018845558167 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4406934678554535 particle/accuracy_1_of_2‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16175709664821625 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.34625616669654846 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4811507761478424 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.028368795290589333 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17394259572029114 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9781359434127808 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0572916679084301 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.222434863448143 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.016081402078270912 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.42862531542778015 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4774138927459717 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009671904146671295 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6987462043762207 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9795918464660645 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0347222238779068 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2838592529296875 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9880763292312622 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9758772850036621 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.48930779099464417 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.3554597795009613 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 (particle/f_score‚0.33636823296546936 jet/accuracy_3_of_7‚0 particle/accuracy_5_of_7‚1 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.40442955493927 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.448875367641449 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.23467431962490082 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.7777777910232544 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.328879714012146 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.301210880279541 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 +jet/accuracy_4_of_5‚0.00358422938734293 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.21913579106330872 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23314207792282104 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.45370370149612427 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16254323720932007 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9616199731826782 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1858760118484497 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.22999414801597595 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17945998907089233 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9861285090446472 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2814815640449524 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2521131932735443 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0034994245506823063 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424243688583374 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18889644742012024 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4822166860103607 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3526749312877655 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3455978035926819 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18625731766223907 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2252417802810669 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5133333206176758 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.18518519401550293 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2504037022590637 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19124244153499603 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995861291885376 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.553510308265686 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5851189494132996 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.39549151062965393 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2002873569726944 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17293399572372437 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5219452977180481 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0297367125749588 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9856705069541931 jet/accuracy_1_of_6‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.997300386428833 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4665505290031433 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.21846845746040344 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9259259104728699 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.41823896765708923 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9892434477806091 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1869659721851349 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5440091490745544 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.22979797422885895 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.035087719559669495 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5964912176132202 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5462963581085205 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5329558849334717 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15485486388206482 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7validation_average_jet_accuracy‚0.40880075097084045 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0436507947742939 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.44827187061309814 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.35164615511894226 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44116270542144775 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7078852653503418 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2209911197423935 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8348867297172546 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014484861865639687 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17324233055114746 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9899402260780334 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 )jet/accuracy_1_of_4‚0.884218692779541 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21759259700775146 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6332427263259888 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 +"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875*‡Ö§ ¤“¦ =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 jet/accuracy_6_of_6‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.972787082195282 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.40442955493927 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.37878790497779846 particle/accuracy_1_of_3‚1 (particle/accuracy‚0.8434503078460693 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29660114645957947 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4228268563747406 +jet/accuracy_3_of_6‚0.03846153989434242 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 _step‚197 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4696170687675476 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.2074723094701767 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9857617020606995 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996565580368042 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23900601267814636 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011022928170859814 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07183908671140671 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5219634175300598 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7651515603065491 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.20133531093597412 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.022641509771347046 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.46162280440330505 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10451014339923859 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04047500342130661 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.410¦‡]þù34483909606934 particle/accuracy_4_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17281650006771088 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10201013833284378 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3536613881587982 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09374584257602692 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333730697632 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9748062491416931 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9887301921844482 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.20588311553001404 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.31651654839515686 +jet/accuracy_2_of_6‚0.46794870495796204 jet/accuracy_1_of_7‚1 particle/accuracy_2_of_7‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24871937930583954 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.483682245016098 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.17688968777656555 +jet/accuracy_3_of_4‚0.06063299998641014 *jet/accuracy_2_of_5‚0.4810164272785187 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39444446563720703 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4103710651397705 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9868301749229431 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20567075908184052 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9150793552398682 particle/accuracy_3_of_7‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03213656321167946 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9837325215339661 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3480391800403595 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9857972860336304 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.400726854801178 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2122131884098053 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2992468774318695 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03896718472242355 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9916955828666687 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4294070303440094 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3143274784088135 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 trainer/global_step‚2417 % _timestamp‚1.7884905333054633e+09 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23457522690296173 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3166666626930237 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1763686239719391 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43798449635505676 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4454023241996765 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833134651184 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3932715654373169 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9956604838371277 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5975695252418518 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2777777910232544 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4864003360271454 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9510416984558105 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_6‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_6_of_7‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37931543588638306 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.39531755447387695 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5705586671829224 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3220538794994354 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.41833844780921936 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5523361563682556 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.38633492588996887 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.39935997128486633 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.984059751033783 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4652778208255768 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20567075908184052 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0384608618915081 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757660865784 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2990013659000397 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02371695265173912 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.995908260345459 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.016599705442786217 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4368303716182709 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3828769624233246 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08040229976177216 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 jet/accuracy_5_of_5‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.416631281375885 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9285714030265808 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04863031208515167 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09047619253396988 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9748587012290955 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9748773574829102 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.40189018845558167 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4406934678554535 particle/accuracy_1_of_2‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16175709664821625 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.34625616669654846 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4811507761478424 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ;"Purity/*FRt*SRqqt3FBt/event_purity‚0.028368795290589333 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17394259572029114 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9781359434127808 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0572916679084301 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.222434863448143 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.016081402078270912 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.42862531542778015 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4774138927459717 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009671904146671295 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6987462043762207 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9795918464660645 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0347222238779068 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2838592529296875 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9880763292312622 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9758772850036621 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.48930779099464417 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.3554597795009613 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 (particle/f_score‚0.33636823296546936 jet/accuracy_3_of_7‚0 particle/accuracy_5_of_7‚1 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.40442955493927 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.448875367641449 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.23467431962490082 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.7777777910232544 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.328879714012146 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.301210880279541 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 +jet/accuracy_4_of_5‚0.00358422938734293 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.21913579106330872 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23314207792282104 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.45370370149612427 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16254323720932007 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9616199731826782 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1858760118484497 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.22999414801597595 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17945998907089233 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9861285090446472 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2814815640449524 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2521131932735443 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0034994245506823063 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424243688583374 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18889644742012024 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4822166860103607 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3526749312877655 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3455978035926819 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18625731766223907 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2252417802810669 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5133333206176758 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.18518519401550293 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2504037022590637 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19124244153499603 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995861291885376 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.553510308265686 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5851189494132996 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.39549151062965393 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2002873569726944 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17293399572372437 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5219452977180481 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0297367125749588 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9856705069541931 jet/accuracy_1_of_6‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.997300386428833 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4665505290031433 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.21846845746040344 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9259259104728699 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.41823896765708923 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9892434477806091 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1869659721851349 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5440091490745544 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.22979797422885895 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.035087719559669495 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5964912176132202 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5462963581085205 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5329558849334717 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15485486388206482 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7validation_average_jet_accuracy‚0.40880075097084045 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0436507947742939 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.44827187061309814 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.35164615511894226 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44116270542144775 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7078852653503418 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2209911197423935 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8348867297172546 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014484861865639687 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17324233055114746 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9899402260780334 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 )jet/accuracy_1_of_4‚0.884218692779541 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21759259700775146 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6332427263259888 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 +"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15777777135372162 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.7777777910232544 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.019592054188251495 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9791848659515381 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1980479210615158 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.36732301115989685 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18330642580986023 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5969685316085815 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.44342759251594543 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3336106538772583 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.33981481194496155 epoch‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 _runtime‚2632.948358242 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1520296037197113 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22268253564834595 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.389428973197937 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.44272997975349426 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4486314356327057 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9596161842346191 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5001489520072937 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.31494253873825073 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9871404767036438 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.378361314535141 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4099999964237213 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.42572179436683655 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19703224301338196 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.42635443806648254 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24945589900016785 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4866733253002167 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.25925925374031067 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.10000000149011612 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.34651413559913635 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.21307794749736786 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_6‚1 particle/accuracy_7_of_7‚0.5 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚43.33010752164224 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4422617256641388 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8148845434188843 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5535079836845398 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0921969786286354 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_5_of_5‚1 jet/accuracy_4_of_6‚0 particle/accuracy_4_of_7‚1 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3215548098087311 particle/accuracy_2_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3977954387664795 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015004695393145084 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936455488204956 ,particle/sensitivity‚0.25428101420402527 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.997300386428833 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4360472857952118 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_4_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285714030265808 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99541836977005 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3342774510383606 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10831031948328018 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 +particle/specificity‚0.9524706602096558 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2777777910232544 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9876121878623962 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 :Šóƒz"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.41396522521972656 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2976190447807312 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.39578917622566223 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.45133933424949646 4 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3928242623806 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04293462634086609 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9920635223388672 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_2_of_4‚0.3653026819229126 *jet/accuracy_3_of_5‚0.0964093804359436 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265ÅO%¶y¥‚(:’ ¬æèÔ毯± gpu.0.gpu‚45.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚163.818' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.763600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õáǽo¦‚(:ä ¬æèÔϬë²% disk./.usageGB‚1174.7992935180664# disk.nvme0c0n1.in‚ 7082.14453125! disk.nvme0n1.in‚ 7082.14453125 disk.dm-0.out‚59465.85546875 disk.sda3.out‚ 79.16015625 disk.sda.in‚0" proc.memory.rssMB‚ 2935.7734375 network.sent‚ 4.940717e+06 disk.dm-0.in‚ 7082.14453125 disk.sda1.out‚0) proc.memory.percent‚17.91853904724121 proc.cpu.threads‚17 disk.dm-1.out‚ 79.16015625 network.recv‚869322% disk.nvme0c0n1.out‚59465.85546875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 cpu‚25.137444210548782) disk./.usagePercent‚78.84176631026048# disk.nvme0n1.out‚59465.85546875 disk.drbd1260.out‚0.359375 disk.sda.out‚ 79.16015625 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚93.15521717071533) proc.memory.availableMB‚ 1121.44921875 disk.dm-1.in‚0ÏÔÀH§‚(:… »æèÔÒý° gpu.0.gpu‚8.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.023 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñãלt¨‚(:é »æèÔ£¦‹² disk.dm-0.in‚ 7082.14453125" proc.memory.rssMB‚ 2935.7734375 cpu‚25.13725247319567 proc.cpu.threads‚17 network.sent‚ 5.063776e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 79.46484375% disk.nvme0n1.out‚60771.6279296875" disk.dm-0.out‚60771.6279296875 disk.sda3.in‚0$ memory_percent‚93.15311908721924 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 7082.14453125 disk.sda3.out‚ 79.46484375) proc.memory.availableMB‚ 1121.79296875) proc.memory.percent‚17.91853904724121) disk./.usagePercent‚78.85827342006077% disk./.usageGB‚1175.0452613830566 disk.drbd1260.in‚0 disk.drbd1260.out‚0.359375 disk.sda.in‚0 disk.sda.out‚ 79.46484375' disk.nvme0c0n1.out‚60771.6279296875! disk.nvme0n1.in‚ 7082.14453125 network.recv‚874408 disk.sda2.out‚0üxùÆ2©- _wandbruntime‚2669  _runtime‚2669÷És›ÇªÁ & lr-AdamW/pg1‚0.0007671255787934228 & lr-AdamW/pg2‚0.0007671255787934228 trainer/global_step‚2449 % _timestamp‚1.7884905698502934e+09 _runtime‚2669.490821568 _step‚1989Q‘%Ì«Æ & lr-AdamW/pg2‚0.0007671255787934228 trainer/global_step‚2449 % _timestamp‚1.7884905698502934e+09 _runtime‚2669.490821568 _step‚198 & lr-AdamW/pg1‚0.0007671255787934228Æ`áR2¬- _wandbruntime‚2669  _runtime‚2669Œàéíí'­çO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2511584420519 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚171614940915200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193174246150656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190541194781696 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _step‚199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38378146 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚2669.492379945 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54420534 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚171614940915200 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54420534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54397519 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97405378 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42984844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24649495 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174136391114240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2632855129088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚171614958918144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚167210585747888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42984842 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚171614958918144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚169722940816815 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚169722153129047 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2520659551744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38378691 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚167210568708528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97382361 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13729194 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42984842 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2512355068927 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 $ _timestamp‚1.788490569851881e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2520659551744 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54397519 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2633051368960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174136391114240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2521432196096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174135600466944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174135600466944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193174025970688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944è%¦‹ò'®ìO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174135600466944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174135600466944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193174025970688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2511584420519 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚171614940915200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193174246150656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190541194781696 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _step‚199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38378146 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚2669.492379945 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54420534 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚171614940915200 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54420534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54397519 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97405378 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42984844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24649495 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174136391114240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2632855129088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚171614958918144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚167210585747888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42984842 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚171614958918144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚169722940816815 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚169722153129047 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2520659551744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38378691 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚167210568708528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97382361 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13729194 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42984842 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2512355068927 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 $ _timestamp‚1.788490569851881e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2520659551744 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54397519 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2633051368960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174136391114240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2521432196096ÇÈ5¨¯‚(:’ ÊæèÔ©œ”° gpu.0.gpu‚34.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.028' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.005599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±þÆt°‚(:é ÊæèÔü§å²% disk.nvme0n1.out‚61018.2490234375" disk.dm-0.out‚61018.2490234375 disk.drbd1260.in‚0 disk.sda.out‚ 79.87890625) proc.memory.percent‚17.91853904724121 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 7082.14453125 disk.dm-0.in‚ 7082.14453125 disk.dm-1.in‚0 disk.dm-1.out‚ 79.87890625 disk.sda1.in‚0$ memory_percent‚93.15314292907715" proc.memory.rssMB‚ 2935.7734375 network.recv‚877696( disk./.usagePercent‚78.8582721400231% disk./.usageGB‚1175.0452423095703 disk.sda3.out‚ 79.87890625 disk.drbd1260.out‚ 0.36328125 disk.sda1.out‚0 cpu‚25.13703596414705 proc.cpu.threads‚17 disk.sda2.in‚0' disk.nvme0c0n1.out‚61018.2490234375! disk.nvme0n1.in‚ 7082.14453125 disk.sda3.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 1121.7890625 network.sent‚ 5.068781e+06.ø2±- _wandbruntime‚2670  _runtime‚2670yÄ™uô²òM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2522436955136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2521664297472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚171682658857472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174205095812608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚169789220704685 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2512585560397 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24659145 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2634100843008 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174205095812608 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43001718 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38393592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54442271 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190616543124480 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43001716 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97420946 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 trainer/global_step‚2449 _runtime‚2670.768035499 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97443989 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97443989 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54419230 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.lõî@arge_pool.freed‚24659143 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2633904616448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38393064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193250643967488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚169790008392661 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚167276635144288 % _timestamp‚1.7884905711267762e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174204305152000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2522436955136 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193250423800832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚167276652183648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚171682640854528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54442271 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43001716 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54419230 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97420946 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13733921 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚171682640854528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚171682658857472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328.–Dý&³÷M J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43001716 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54419230 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97420946 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13733921 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚171682640854528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚171682658857472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2522436955136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2521664297472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚171682658857472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174205095812608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚169789220704685 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2512585560397 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24659145 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2634100843008 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174205095812608 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43001718 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38393592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54442271 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190616543124480 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43001716 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97420946 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 trainer/global_step‚2449 _runtime‚2670.768035499 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97443989 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97443989 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54419230 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24659143 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2633904616448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38393064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193250643967488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚169790008392661 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚167276635144288 % _timestamp‚1.7884905711267762e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174204305152000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2522436955136 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193250423800832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚167276652183648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚171682640854528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54442271 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ÈŒ·«'2´- _wandbruntime‚2670  _runtime‚2670Âñ‚¡[µÕ % _timestamp‚1.7884905711297095e+09 0loss/FRt3/assignment_loss‚2.6540167234381853 1loss/SRqqt2/assignment_loss‚2.150119012221694 1loss/FBt1/assignment_loss‚0.31837299149483445 .loss/FRt4/detection_loss‚1.590014366250412 trainer/global_step‚2449 %loss/total_loss‚43.74623666241806 /loss/FRt2/detection_loss‚2.3946373239159584 2loss/SRqqt3/assignment_loss‚1.1120959349921835 3loss/FBt3/assignment_loss‚0.0064649214968085286 /loss/FRt1/detection_loss‚2.4436869898980316 0loss/SRqqt3/detection_loss‚1.619250220816527 epoch‚2 2loss/SRqqt4/assignment_loss‚0.9873503267765045 .loss/FBt3/detection_loss‚3.312171125411987 /loss/FBt4/detection_loss‚0.5062478360275484 0loss/FRt1/assignment_loss‚3.7083238498731093 0loss/SRqqt2/detection_loss‚2.328151948750019 /loss/FBt2/detection_loss‚2.7177904725074766 _step‚201 .loss/FRt2/assignment_loss‚3.78435081243515 1loss/FBt2/assignment_loss‚0.12183534647338093 0loss/FBt4/assignment_loss‚0.1491487885299298 .loss/FBt1/detection_loss‚2.747704815864563 0loss/FRt4/assignment_loss‚1.7549443607236825 /loss/FRt3/detection_loss‚2.2319037677073967 0loss/SRqqt1/detection_loss‚2.362686453443585 _runtime‚2670.769953644 2loss/SRqqt1/assignment_loss‚1.6842001867339467 1loss/SRqqt4/detection_loss‚1.0607680866351494¥K<º`¶Ú 0loss/FRt3/assignment_loss‚2.6540167234381853 1loss/SRqqt2/assignment_loss‚2.150119012221694 1loss/FBt1/assignment_loss‚0.31837299149483445 .loss/FRt4/detection_loss‚1.590014366250412 trainer/global_step‚2449 %loss/total_loss‚43.74623666241806 /loss/FRt2/detection_loss‚2.3946373239159584 2loss/SRqqt3/assignment_loss‚1.1120959349921835 3loss/FBt3/assignment_loss‚0.0064649214968085286 /loss/FRt1/detection_loss‚2.4436869898980316 0loss/SRqqt3/detection_loss‚1.619250220816527 epoch‚2 2loss/SRqqt4/assignment_loss‚0.9873503267765045 .loss/FBt3/detection_loss‚3.312171125411987 /loss/FBt4/detection_loss‚0.5062478360275484 0loss/FRt1/assignment_loss‚3.7083238498731093 0loss/SRqqt2/detection_loss‚2.328151948750019 /loss/FBt2/detection_loss‚2.7177904725074766 _step‚201 .loss/FRt2/assignment_loss‚3.78435081243515 1loss/FBt2/assignment_loss‚0.12183534647338093 0loss/FBt4/assignment_loss‚0.1491487885299298 .loss/FBt1/detection_loss‚2.747704815864563 0loss/FRt4/assignment_loss‚1.7549443607236825 /loss/FRt3/detection_loss‚2.2319037677073967 0loss/SRqqt1/detection_loss‚2.362686453443585 _runtime‚2670.769953644 2loss/SRqqt1/assignment_loss‚1.6842001867339467 1loss/SRqqt4/detection_loss‚1.0607680866351494 % _timestamp‚1.7884905711297095e+09ÉÕ»’·‚(:‡ ÙæèÔÓéÛ® gpu.0.gpu‚49.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9564 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’z(Ju¸‚(:ê ÙæèԪ㯠disk.dm-1.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚61457.64990234375 disk.dm-0.in‚ 7082.14453125 proc.cpu.threads‚17 network.recv‚884196 disk.drbd1260.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚61457.64990234375! disk.nvme0n1.in‚ 7082.14453125# disk.dm-0.out‚61457.64990234375( proc.memory.availableMB‚ 1121.7578125% disk./.usageGB‚1175.0217742919922 disk.sda3.in‚0 disk.drbd1260.out‚ 0.36328125 disk.sda1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 7082.14453125 cpu‚25.13698189703477 network.sent‚ 5.166447e+06 disk.dm-1.out‚ 80.3359375 disk.sda.in‚0 disk.sda.out‚ 80.3359375$ memory_percent‚93.15333366394043" proc.memory.rssMB‚ 2935.7734375) proc.memory.percent‚17.91853904724121) disk./.usagePercent‚78.85669718167246 disk.sda3.out‚ 80.3359375:\J¤’¹‚(:‡ èæèÔøÂÙ® gpu.0.gpu‚36.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.819' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5638 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦úVº‚(:Ë èæèÔ•â± disk.drbd1260.out‚ 0.37109375" proc.memory.rssMB‚ 2935.7734375) proc.memory.percent‚17.91853904724121 proc.cpu.threads‚17' disk.nvme0c0n1.out‚61872.8173828125% disk.nvme0n1.out‚61872.8173828125 disk.sda.in‚0 disk.sda.out‚80.75 disk.sda2.in‚0 disk.dm-0.in‚7216.875" disk.dm-0.out‚61872.8173828125 disk.sda3.in‚0 network.recv‚889980 disk.sda3.out‚80.75 disk.dm-1.out‚80.75$ memory_percent‚93.15335750579834 cpu‚25.136745632814833 network.sent‚ 5.264232e+06) disk./.usagePercent‚78.85669871771765% disk./.usageGB‚1175.0217971801758 disk.sda1.in‚0 disk.nvme0c0n1.in‚7216.875 disk.drbd1260.in‚0) proc.memory.availableMB‚ 1121.75390625 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚7216.875 disk.dm-1.in‚0~æ>e’»‚(:‡ ÷æèÔ«÷¼® gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.066' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ °m¼‚(:â ÷æèÔ…µó± disk.sda.out‚ 81.0390625& disk.nvme0c0n1.out‚62341.927734375 disk.dm-0.in‚ 7222.2421875) proc.memory.availableMB‚ 1121.75390625) proc.memory.percent‚17.91853904724121% disk./.usageGB‚1175.5217781066895 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.out‚ 81.0390625 disk.sda1.out‚0 disk.sda2.in‚0! disk.dm-0.out‚62341.927734375" proc.memory.rssMB‚ 2935.7734375 cpu‚25.13663537379817 network.recv‚893404 disk.sda3.out‚ 81.0390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.37109375" disk.nvme0c0n1.in‚ 7222.2421875 disk.nvme0n1.in‚ 7222.2421875$ disk.nvme0n1.out‚62341.927734375$ memory_percent‚93.15335750579834 network.sent‚ 5.269365e+06) disk./.usagePercent‚78.89025285720793 disk.dm-1.in‚0f ù>2½- _wandbruntime‚2719  _runtime‚2719ÍA±?ǾÁ & lr-AdamW/pg1‚0.0007650831450547015 & lr-AdamW/pg2‚0.0007650831450547015 trainer/global_step‚2499 % _timestamp‚1.7884906196710732e+09 _runtime‚2719.311486974 _step‚202ïYÊÌ¿Æ & lr-AdamW/pg1‚0.0007650831450547015 & lr-AdamW/pg2‚0.0007650831450547015 trainer/global_step‚2499 % _timestamp‚1.7884906196710732e+09 _runtime‚2719.311486974 _step‚202Êáh=2À- _wandbruntime‚2719  _runtime‚2719ù.òî'ÁèO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2571683896320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2684972936192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177570849133568 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2685177564672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196993789485568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39124191 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25131895 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13992294 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194308611920896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2561655005627 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99337980 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚177570849133568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173075545502155 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55486421 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173076333189923 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174999955884544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204628480 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43828544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174999955884544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚196993560916992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0M[HÈÂíO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194308587980800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55509436 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174999937881600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39123642 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55486421 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177570849133568 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2685177564672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196993789485568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocïÝÌú+!ated‚39124191 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25131895 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13992294 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194308611920896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2561655005627 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99337980 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚177570849133568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173075545502155 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55486421 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173076333189923 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174999955884544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204628480 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43828544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174999955884544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚196993560916992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2571683896320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2684972936192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99376591 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99353548 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚175067655823872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2572688655360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43845416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25141545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194383936323584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173143400765769 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚175067637820928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2571915997696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43845418 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197069947227136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55531173 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177640344479232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39138404 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2686010903552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173142613077793 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99353548 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2686213421568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13996861 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202518016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884906206443038e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39138938 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2572688655360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2571915997696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25141543 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99376591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43845418 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226458112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177639553818624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚175067655823872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55531173 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2563426794121 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 trainer/global_step‚2499 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197070173685248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114n¦û«ý&Å÷M N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99353548 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2686213421568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13996861 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202518016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884906206443038e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39138938 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2572688655360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2571915997696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25141543 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99376591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43845418 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226458112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177639553818624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚175067655823872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55531173 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2563426794121 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 trainer/global_step‚2499 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197070173685248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99376591 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99353548 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚175067655823872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2572688655360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43845416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25141545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194383936323584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173143400765769 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚175067637820928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2571915997696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43845418 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197069947227136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55531173 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177640344479232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39138404 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2686010903552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173142613077793 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664Ì‹ y2Æ- _wandbruntime‚2720  _runtime‚2720¤Aº½YÇÓ 0loss/FRt1/assignment_loss‚3.4054891847901874 /loss/FRt2/assignment_loss‚3.750166315020937 .loss/FRt1/detection_loss‚2.581754970881674 0loss/SRqqt1/detection_loss‚2.140320094994136 /loss/FBt1/detection_loss‚3.1658351288901434 epoch‚2 _runtime‚2720.287515377 1loss/FBt1/assignment_loss‚0.36138378445886904 .loss/FRt3/detection_loss‚2.062005161362536 .loss/FRt4/detection_loss‚1.811076017362731 2loss/SRqqt3/assignment_loss‚1.5650353582632552 1loss/FBt4/assignment_loss‚0.11631051726628662 /loss/FBt2/detection_loss‚3.4452386647462845 /loss/FRt3/assignment_loss‚2.749581983276442 /loss/FBt3/detection_loss‚2.6986906907775183 /loss/FBt4/detection_loss‚0.5166921119107619 0loss/FRt4/assignment_loss‚1.3443519685949599 1loss/FBt2/assignment_loss‚0.24679565582118812 1loss/FBt3/assignment_loss‚0.04781057993585074 1loss/SRqqt3/detection_loss‚1.5341872389654856 0loss/SRqqt4/detection_loss‚1.106237591620077 2loss/SRqqt1/assignment_loss‚1.4242892454777445 2loss/SRqqt2/assignment_loss‚1.5929969501448795 /loss/FRt2/detection_loss‚2.5477343147451226 /loss/SRqqt2/detection_loss‚2.55748650431633 trainer/global_step‚2499 1loss/SRqqt4/assignment_loss‚0.733761169539209 %loss/total_loss‚43.50523120316261 % _timestamp‚1.7884906206472719e+09 _step‚205×ÿ¹^^ÈØ 2loss/SRqqt2/assignment_loss‚1.5929969501448795 /loss/FRt2/detection_loss‚2.5477343147451226 /loss/SRqqt2/detection_loss‚2.55748650431633 trainer/global_step‚2499 1loss/SRqqt4/assignment_loss‚0.733761169539209 %loss/total_loss‚43.50523120316261 % _timestamp‚1.7884906206472719e+09 _step‚205 0loss/FRt1/assignment_loss‚3.4054891847901874 /loss/FRt2/assignment_loss‚3.750166315020937 .loss/FRt1/detection_loss‚2.581754970881674 0loss/SRqqt1/detection_loss‚2.140320094994136 /loss/FBt1/detection_loss‚3.1658351288901434 epoch‚2 _runtime‚2720.287515377 1loss/FBt1/assignment_loss‚0.36138378445886904 .loss/FRt3/detection_loss‚2.062005161362536 .loss/FRt4/detection_loss‚1.811076017362731 2loss/SRqqt3/assignment_loss‚1.5650353582632552 1loss/FBt4/assignment_loss‚0.11631051726628662 /loss/FBt2/detection_loss‚3.4452386647462845 /loss/FRt3/assignment_loss‚2.749581983276442 /loss/FBt3/detection_loss‚2.6986906907775183 /loss/FBt4/detection_loss‚0.5166921119107619 0loss/FRt4/assignment_loss‚1.3443519685949599 1loss/FBt2/assignment_loss‚0.24679565582118812 1loss/FBt3/assignment_loss‚0.04781057993585074 1loss/SRqqt3/detection_loss‚1.5341872389654856 0loss/SRqqt4/detection_loss‚1.106237591620077 2loss/SRqqt1/assignment_loss‚1.4242892454777445Íã\+>É‚(:’ †çèÔŸëÄ® gpu.0.gpu‚40.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚132.399' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.479799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EиMkÊ‚(:à †çèÔÞ‰í¯ disk.sda1.in‚0 disk.nvme0n1.in‚ 7222.2421875 disk.drbd1260.in‚0( proc.memory.availableMB‚ 1121.7109375" proc.memory.rssMB‚ 2935.7734375 cpu‚25.136477255540225 network.recv‚896870 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 81.48046875 disk.dm-1.in‚0 disk.drbd1260.out‚0.375 disk.sda.out‚ 81.48046875$ memory_percent‚93.15361976623535! disk.dm-0.out‚62660.728515625 network.sent‚ 5.274363e+06 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 7222.2421875& disk.nvme0c0n1.out‚62660.728515625$ disk.nvme0n1.out‚62660.728515625 disk.dm-0.in‚ 7222.2421875 disk.sda.in‚0) disk./.usagePercent‚78.89182781555857% disk./.usageGB‚1175.5452461242676 disk.sda1.out‚0 disk.dm-1.out‚ 81.48046875) proc.memory.percent‚17.91853904724121 proc.cpu.threads‚17mo,éwË‚(:‡ •çèǪ̂ԭ gpu.0.gpu‚27.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0' á.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.748' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥otjÌ‚(:ß •çèÔü“æ¯ disk.dm-1.out‚ 82.01953125( proc.memory.availableMB‚ 1121.7109375 proc.cpu.threads‚17 network.recv‚903710) disk./.usagePercent‚78.89182627951337 disk.sda2.in‚0! disk.dm-0.out‚63027.490234375 disk.dm-1.in‚0) proc.memory.percent‚17.91853904724121 network.sent‚ 5.383209e+06 disk.sda.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚63027.490234375 disk.nvme0n1.in‚ 7222.2421875 disk.drbd1260.out‚0.375" proc.memory.rssMB‚ 2935.7734375 cpu‚25.136196245513048 disk.sda.out‚ 82.01953125 disk.sda1.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚63027.490234375 disk.sda3.in‚0 disk.sda3.out‚ 82.01953125 disk.drbd1260.in‚0$ memory_percent‚93.15361976623535$ disk./.usageGB‚1175.545223236084" disk.nvme0c0n1.in‚ 7222.2421875 disk.dm-0.in‚ 7222.2421875Ÿ n’Í‚(:‡ ¤çèÔü¶ô¯ gpu.0.gpu‚24.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9564 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãÒÿm΂(:â ¤çèԿƺ² disk.dm-1.out‚ 82.5234375 disk.drbd1260.in‚0" proc.memory.rssMB‚ 2935.7734375) proc.memory.percent‚17.91853904724121 disk.sda1.out‚0) disk./.usagePercent‚78.89182755955105% disk./.usageGB‚1175.5452423095703 disk.sda3.in‚0 disk.sda1.in‚0$ disk.nvme0n1.out‚63274.302734375 cpu‚25.136132696276793 disk.sda3.out‚ 82.5234375 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 7222.2421875& disk.nvme0c0n1.out‚63274.302734375! disk.dm-0.out‚63274.302734375$ memory_percent‚93.15381050109863 proc.cpu.threads‚17 network.sent‚ 5.389058e+06 disk.drbd1260.out‚ 0.37890625 disk.sda.in‚0 disk.sda.out‚ 82.5234375 disk.sda2.in‚0 disk.nvme0n1.in‚ 7222.2421875 disk.dm-0.in‚ 7222.2421875( proc.memory.availableMB‚ 1121.6796875 network.recv‚907867 disk.dm-1.in‚00±‚µ2Ï- _wandbruntime‚2771  _runtime‚2771®âœ{ÆÐÀ $ _timestamp‚1.788490671749037e+09 _runtime‚2771.389440898 _step‚206 & lr-AdamW/pg1‚0.0007629638201692441 & lr-AdamW/pg2‚0.0007629638201692441 trainer/global_step‚2549$‚Y,ËÑÅ & lr-AdamW/pg1‚0.0007629638201692441 & lr-AdamW/pg2‚0.0007629638201692441 trainer/global_step‚2549 $ _timestamp‚1.788490671749037e+09 _runtime‚2771.389440898 _step‚206ÎÏ™qS2Ò- _wandbruntime‚2771  _runtime‚2771äE*ñ'ÓëO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44672242 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚178384952850944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176428937875263 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44672244 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44672244 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200813127835648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2611725590735 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181006097800192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178384934848000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181006888447488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2621935596544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198076029060096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173817212284528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101270582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 trainer/global_step‚2549 _runtime‚2771.390616995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25614297 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44672242 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56598338 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178384934848000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56575323 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2737321052672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2612496239143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25614295 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2621935596544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2621162952192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚178384952850944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39868318 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198076005120000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 $ _timestamp‚1.788490671750162e+09 _step‚207 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56575323 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181006888447488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2737122715648Bygãö'ÔðO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56598338 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178384934848000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56575323 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2737321052672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2612496239143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25614295 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2621935596544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2621162952192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚178384952850944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39868318 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198076005120000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 $ _timestamp‚1.788490671750162e+09 _step‚207 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56575323 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181006888447488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2737122715648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44672242 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚178384952850944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176428937875263 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44672244 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44672244 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200813127835648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2611725590735 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181006097800192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178384934848000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181006888447488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2621935596544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198076029060096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173817212284528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101270582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 trainer/global_step‚2549 _runtime‚2771.390616995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25614297 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44672242 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992ÏDD»2Õ- _wandbruntime‚2772  _runtime‚2772Þ\r($ÖöM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚176496005450901 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173883295759648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 % _timestamp‚1.7884906726042895e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44689116 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232749568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208809472 trainer/global_step‚2549 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200889505128448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2738360475136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44689118 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181075593145856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25623945 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101309193 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2738151665664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2622167697920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44689116 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101286150 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56597034 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚176496793138877 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173883278720288 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56620075 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198151377402880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101309193 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2613497379229 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39883189 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2612726730613 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178452652790272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚178452634787328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39883736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 YGDeviceStatsMonitor.on_train_batch_ÍEc·âend/active_bytes.small_pool.allocated‚ 2622940355584 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181074802485248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚178452634787328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14259791 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198151353462784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0Ÿ'×ûM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200889737878016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _runtime‚2772.245536513 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44689118 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56620075 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2622167697920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44689116 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101286150 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56597034 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚176496793138877 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173883278720288 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56620075 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198151377402880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101309193 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2613497379229 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39883189 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2612726730613 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178452652790272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚178452634787328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39883736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2622940355584 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181074802485248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚178452634787328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14259791 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198151353462784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚176496005450901 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173883295759648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 % _timestamp‚1.7884906726042895e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44689116 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232749568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208809472 trainer/global_step‚2549 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200889505128448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2738360475136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44689118 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181075593145856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25623945 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101309193 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2738151665664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184441346466816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181769949817344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚179783117936139 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202531328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2789446524928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103203184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57664225 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2789243993600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2671414652416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45515944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚184441346466816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2662566824251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚211 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚184442137114112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103180167 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57687240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26096697 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204632666252800 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177120534072528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2671414652416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328ùÚóøÆæñO trainer/global_step‚2599 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2672187296768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2672187296768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚184442137114112 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57687240 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226471424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201843446199296 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40613668 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201843422259200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _runtime‚2817.821901979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40614191 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14517494 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 % _timestamp‚1.7884907181808903e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26096695 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2662566824251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚211 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚184442137114112 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103180167 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57687240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26096697 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204632666252800 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177120534072528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2671414652416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184441346466816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181769949817344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚179783117936139 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202531328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2789446524928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103203184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57664225 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2789243993600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2671414652416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45515944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚184441346466816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14522712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2599 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26106345 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181837649756672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2672419398144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26106343 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181837649756672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204709050320384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚181837631753728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179849397824009 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201918794542080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚177186600508288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14522180 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2790279718400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57708977 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2673192055808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45532816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204709278875648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57685936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚181837631753728 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45532816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2672419398144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103218752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103241795 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40629057 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40628523 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228555264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2663567964337 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103241795 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204615168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2662797315721 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14522712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2599 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26106345 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181837649756672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2672419398144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26106343 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181837649756672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204709050320384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚181837631753728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179849397824009 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201918794542080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚177186600508288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14522180 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2790279718400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57708977 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2673192055808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45532816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204709278875648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57685936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚181837631753728 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45532816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2672419398144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103218752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103241795 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40629057 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40628523 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228555264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2663567964337 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103241795 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204615168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2662797315721 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 Fæv2ê- _wandbruntime‚2818  _runtime‚2818|t‰]WëÑ .loss/FRt2/detection_loss‚2.832397270947695 .loss/FBt1/detection_loss‚2.929537817835808 2loss/SRqqt1/assignment_loss‚1.6880746387848349 0loss/SRqqt2/detection_loss‚2.191095219208644 .loss/FBt2/detection_loss‚2.400065226988359 epoch‚2 $ _timestamp‚1.788490719193831e+09 0loss/FRt4/assignment_loss‚1.4336345207016423 2loss/SRqqt3/assignment_loss‚1.5832721725746524 1loss/FBt4/assignment_loss‚0.12552382156715075 .loss/FRt1/detection_loss‚2.742757646661056 /loss/FRt4/detection_loss‚1.5641631164921905 1loss/SRqqt3/detection_loss‚1.5138594654854387 /loss/FBt3/detection_loss‚2.9276892890532813 /loss/FBt4/detection_loss‚0.6346754605130461 2loss/SRqqt2/assignment_loss‚2.2563084923686128 _runtime‚2818.834021863 _step‚213 1loss/FBt1/assignment_loss‚0.06597792590036988 trainer/global_step‚2599 0loss/FRt1/assignment_loss‚3.4543132734926125 /loss/FRt2/assignment_loss‚3.826132945716381 1loss/SRqqt4/assignment_loss‚0.664115278981626 1loss/FBt2/assignment_loss‚0.11632955686284484 .loss/FRt3/detection_loss‚2.208848067692348 0loss/FRt3/assignment_loss‚2.8510558945792064 /loss/FBt3/assignment_loss‚0.206167091455427 0loss/SRqqt1/detection_loss‚2.478775309793877 1loss/SRqqt4/detection_loss‚0.8443147735114196 %loss/total_loss‚43.53908427716853Án¼\ìÖ /loss/FBt4/detection_loss‚0.6346754605130461 2loss/SRqqt2/assignment_loss‚2.2563084923686128 _runtime‚2818.834021863 _step‚213 1loss/FBt1/assignment_loss‚0.06597792590036988 trainer/global_step‚2599 0loss/FRt1/assignment_loss‚3.4543132734926125 /loss/FRt2/assignment_loss‚3.826132945716381 1loss/SRqqt4/assignment_loss‚0.664115278981626 1loss/FBt2/assignment_loss‚0.11632955686284484 .loss/FRt3/detection_loss‚2.208848067692348 0loss/FRt3/assignment_loss‚2.8510558945792064 /loss/FBt3/assignment_loss‚0.206167091455427 0loss/SRqqt1/detection_loss‚2.478775309793877 1loss/SRqqt4/detection_loss‚0.8443147735114196 %loss/total_loss‚43.53908427716853 .loss/FRt2/detection_loss‚2.832397270947695 .loss/FBt1/detection_loss‚2.929537817835808 2loss/SRqqt1/assignment_loss‚1.6880746387848349 0loss/SRqqt2/detection_loss‚2.191095219208644 .loss/FBt2/detection_loss‚2.400065226988359 epoch‚2 $ _timestamp‚1.788490719193831e+09 0loss/FRt4/assignment_loss‚1.4336345207016423 2loss/SRqqt3/assignment_loss‚1.5832721725746524 1loss/FBt4/assignment_loss‚0.12552382156715075 .loss/FRt1/detection_loss‚2.742757646661056 /loss/FRt4/detection_loss‚1.5641631164921905 1loss/SRqqt3/detection_loss‚1.5138594654854387 /loss/FBt3/detection_loss‚2.9276892890532813Õ¥<ší‚(:… àçèÔö†ä¬ gpu.0.gpu‚33.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.46' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎbÖxî‚(:í àçèÔҞȰ disk.dm-1.out‚ 84.19921875 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚93.15714836120605( disk./.usagePercent‚78.7933788382288( disk.nvme0c0n1.out‚64869.38818359375! disk.nvme0n1.in‚ 7367.72265625 disk.dm-0.in‚ 7367.72265625 disk.dm-1.in‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2935.7734375 proc.cpu.threads‚17 disk.sda.out‚ 84.19921875( proc.memory.availableMB‚ 1121.1328125) proc.memory.percent‚17.91853904724121 cpu‚25.135160232041553 network.sent‚ 5.512948e+06# disk.nvme0c0n1.in‚ 7367.72265625& disk.nvme0n1.out‚64869.38818359375 disk.sda3.out‚ 84.19921875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.38671875 disk.sda1.out‚0 disk.sda2.out‚0 network.recv‚924695% disk./.usageGB‚1174.0782852172852# disk.dm-0.out‚64869.38818359375 disk.sda3.in‚0@|›´ï‚(:‚ ïçèÔ󕙯 gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚157.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;¥ú¯pð‚(:å ïçèԮƼ± network.sent‚ 5.620616e+06 disk.sda3.out‚84.53125 disk.dm-1.out‚84.53125 disk.sda.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 7521.71484375& disk.nvme0n1.out‚65187.28271484375$ disk./.usageGB‚1173.622428894043# disk.nvme0c0n1.in‚ 7521.71484375# disk.dm-0.out‚65187.28271484375$ memory_percent‚93.15426349639893 proc.cpu.threads‚17 network.recv‚930487) disk./.usagePercent‚78.76278593788709 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.38671875( disk.nvme0c0n1.out‚65187.28271484375) proc.memory.percent‚17.91853904724121 disk.sda.out‚84.53125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 7521.71484375) proc.memory.availableMB‚ 1121.60546875" proc.memory.rssMB‚ 2935.7734375 cpu‚25.134791710105436›Å’ñ‚(:‡ þçèÔªÞ´® gpu.0.gpu‚44.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.699' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1398 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E ô|tò‚(:é þçèÔ¤Ëü±) proc.memory.availableMB‚ 1121.59765625" proc.memory.rssMB‚ 2935.7734375 network.recv‚933958% disk./.usageGB‚1173.5989418029785 disk.sda.out‚ 84.8671875 disk.sda1.in‚0 disk.dm-0.in‚ 7521.71484375) proc.memory.percent‚17.91853904724121 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0# disk.dm-0.out‚65391.21923828125 cpu‚25.13460528666579 network.sent‚ 5.625676e+06) disk./.usagePercent‚78.76120969949876 disk.sda3.out‚ 84.8671875 disk.dm-1.in‚0 disk.dm-1.out‚ 84.8671875 disk.drbd1260.out‚0.390625 disk.sda1.out‚0( disk.nvme0c0n1.out‚65391.21923828125 disk.sda.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 7521.71484375! disk.nvme0n1.in‚ 7521.71484375& disk.nvme0n1.out‚65391.21923828125$ memory_percent‚93.15431118011475UÁÀ<2ó- _wandbruntime‚2864  _runtime‚2864cƒeÆôÀ & lr-AdamW/pg2‚0.0007584962795465976 trainer/global_step‚2649 $ _timestamp‚1.788490765006646e+09 _runtime‚2864.647032501 _step‚214 & lr-AdamW/pg1‚0.0007584962795465976l××HõÅ _runtime‚2864.647032501 _step‚214 & lr-AdamW/pg1‚0.0|U›Þƒ007584962795465976 & lr-AdamW/pg2‚0.0007584962795465976 trainer/global_step‚2649 $ _timestamp‚1.788490765006646e+09ÖÉXkã2ö- _wandbruntime‚2864  _runtime‚2864ŸéØoñ'÷ëO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚183136510309247 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚185154946783744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚185154928780800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105135786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14782876 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180423872899888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205610863338496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58753127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14782344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187877385780736 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _runtime‚ 2864.64852781 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46359644 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2721666352640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105112769 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚185154946783744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2711866760951 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46359644 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2722438996992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46359642 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210919936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234860032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2841885952512 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187876595133440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208452749291008 trainer/global_step‚2649 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58753127 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2722438996992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46359642 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205610839398400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2841675032576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960ûy¨ö'øðO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180423872899888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205610863338496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58753127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14782344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187877385780736 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _runtime‚ 2864.64852781 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46359644 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2721666352640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105112769 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚185154946783744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2711866760951 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46359644 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2722438996992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46359642 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210919936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234860032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2841885952512 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187876595133440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208452749291008 trainer/global_step‚2649 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58753127 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2722438996992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46359642 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205610839398400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2841675032576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚183136510309247 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚185154946783744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚185154928780800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105135786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14782876 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0×#^ŸÛù‚(:’ èèÔü–œ­ gpu.0.gpu‚26.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.167' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.833400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Eàþesú‚(:è èèÔ’¢ô¯% disk./.usageGB‚1172.5300903320312 disk.sda.in‚0 disk.sda.out‚ 85.38671875# memory_percent‚93.1558609008789 cpu‚25.13429531810685 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 85.38671875 disk.drbd1260.out‚0.390625 disk.sda1.in‚0! disk.nvme0n1.in‚ 7521.71484375" proc.memory.rssMB‚ 2935.7734375 network.sent‚ 5.631959e+06 network.recv‚938719) disk./.usagePercent‚78.68947818045737 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 7521.71484375( disk.nvme0c0n1.out‚65679.03173828125& disk.nvme0n1.out‚65679.03173828125& proc.memory.availableMB‚ 1121.34375 disk.sda3.out‚ 85.38671875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 7521.71484375# disk.dm-0.out‚65679.03173828125) proc.memory.percent‚17.91853904724121 proc.cpu.threads‚17›§G2û- _wandbruntime‚2865  _runtime‚2865}x)ü&üöM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46376518 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105174397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41376340 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2842706374656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58774838 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2723443756032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205686187741184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚538 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2722671098368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2842915184128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185222628720128 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2723443756032 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 % _timestamp‚1.7884907659165018e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46376516 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180489939335648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58797879 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208528894115840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58774838 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58797879 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14787597 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚2865.557318048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0…Y öçýûM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2723443756032 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 % _timestamp‚1.7884907659165018e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 J=DeviceStatsMonitor.on_train_bat< ¤$ch_end/active.large_pool.freed‚46376516 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180489939335648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58797879 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208528894115840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58774838 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58797879 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14787597 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚2865.557318048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46376518 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105174397 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41376340 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2842706374656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58774838 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2723443756032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205686187741184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚538 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2722671098368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2842915184128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185222628720128 BìP2þ- _wandbruntime‚2865  _runtime‚2865ù:/YÿÓ %loss/total_loss‚45.40860115739491 2loss/SRqqt1/assignment_loss‚1.4682289253215532 0loss/FBt2/assignment_loss‚0.4108356349170208 1loss/FBt4/assignment_loss‚0.12130410095384292 .loss/FRt1/detection_loss‚2.394062461500818 /loss/FBt3/detection_loss‚2.8204711437225343 /loss/FRt3/detection_loss‚2.7414345056624025 .loss/FBt1/detection_loss‚2.098695857184274 _step‚217 /loss/FRt2/detection_loss‚2.4716766743283523 1loss/SRqqt3/detection_loss‚1.7454516934230924 0loss/FRt4/assignment_loss‚2.0894737687765383 /loss/FBt4/detection_loss‚0.5659911972863673 trainer/global_step‚2649 /loss/FRt2/assignment_loss‚2.743608186119481 2loss/SRqqt2/assignment_loss‚1.5451895639300346 1loss/FBt3/assignment_loss‚0.15966776754357853 /loss/FRt4/detection_loss‚1.8593077544561203 .loss/FBt2/detection_loss‚4.039134532213211 $ _timestamp‚1.788490765919093e+09 0loss/FRt1/assignment_loss‚3.1953150182962418 1loss/SRqqt3/assignment_loss‚1.596259958751034 2loss/SRqqt4/assignment_loss‚1.0675511012474697 0loss/SRqqt1/detection_loss‚2.323500806415403 1loss/SRqqt2/detection_loss‚2.4716370244820913 /loss/FBt1/assignment_loss‚0.805367727896997 _runtime‚2865.559280367 0loss/FRt3/assignment_loss‚3.4431407951019906 1loss/SRqqt4/detection_loss‚1.2312949578644652 epoch‚2åö<_^€Ø 0loss/FRt3/assignment_loss‚3.4431407951019906 1loss/SRqqt4/detection_loss‚1.2312949578644652 epoch‚2 %loss/total_loss‚45.40860115739491 2loss/SRqqt1/assignment_loss‚1.4682289253215532 0loss/FBt2/assignment_loss‚0.4108356349170208 1loss/FBt4/assignment_loss‚0.12130410095384292 .loss/FRt1/detection_loss‚2.394062461500818 /loss/FBt3/detection_loss‚2.8204711437225343 /loss/FRt3/detection_loss‚2.7414345056624025 .loss/FBt1/detection_loss‚2.098695857184274 _step‚217 /loss/FRt2/detection_loss‚2.4716766743283523 1loss/SRqqt3/detection_loss‚1.7454516934230924 0loss/FRt4/assignment_loss‚2.0894737687765383 /loss/FBt4/detection_loss‚0.5659911972863673 trainer/global_step‚2649 /loss/FRt2/assignment_loss‚2.743608186119481 2loss/SRqqt2/assignment_loss‚1.5451895639300346 1loss/FBt3/assignment_loss‚0.15966776754357853 /loss/FRt4/detection_loss‚1.8593077544561203 .loss/FBt2/detection_loss‚4.039134532213211 $ _timestamp‚1.788490765919093e+09 0loss/FRt1/assignment_loss‚3.1953150182962418 1loss/SRqqt3/assignment_loss‚1.596259958751034 2loss/SRqqt4/assignment_loss‚1.0675511012474697 0loss/SRqqt1/detection_loss‚2.323500806415403 1loss/SRqqt2/detection_loss‚2.4716370244820913 /loss/FBt1/assignment_loss‚0.805367727896997 _runtime‚2865.559280367ÙÀ˜´…Ž‚(:ƒ œèèÔëî™® gpu.0.gpu‚36.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.1' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.62 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î¥îxx‚‚(:í œèèÔÐæ¯ proc.cpu.threads‚17 network.sent‚ 5.729037e+06 network.recv‚944841 disk.sda.in‚0 disk.sda.out‚ 85.82421875 disk.sda2.in‚0( disk.nvme0c0n1.out‚66046.00927734375$ memory_percent‚93.15474033355713" proc.memory.rssMB‚ 2935.7734375 disk.dm-0.in‚ 7521.71484375# disk.dm-0.out‚66046.00927734375 disk.dm-1.in‚0 disk.dm-1.out‚ 85.82421875 disk.drbd1260.out‚ 0.39453125 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 7521.71484375) proc.memory.availableMB‚ 1121.52734375) disk./.usagePercent‚78.69102472197173& disk.nvme0n1.out‚66046.00927734375 disk.sda3.in‚0 disk.sda3.out‚ 85.82421875 disk.drbd1260.in‚0) proc.memory.percent‚17.91853904724121$ disk./.usageGB‚1172.553134918213! disk.nvme0n1.in‚ 7521.71484375 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.13404751331485 Pþ^’ƒ‚(:‡ «èèÔÓ—¾° gpu.0.gpu‚33.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.994' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âžë™y„‚(:î «èèÔ½–Ž² disk.sda1.out‚0( disk.nvme0c0n1.out‚66309.59912109375# disk.dm-0.out‚66309.59912109375 disk.sda3.out‚ 86.19140625) proc.memory.percent‚17.91853904724121% disk./.usageGB‚1172.5296516418457 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 7696.32421875 disk.dm-1.in‚0) disk./.usagePercent‚78.68944873959094 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 7696.32421875 disk.dm-1.out‚ 86.19140625$ memory_percent‚93.15485954284668" proc.memory.rssMB‚ 2935.7734375 proc.cpu.threads‚17 network.recv‚950070 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 7696.32421875& disk.nvme0n1.out‚66309.59912109375 disk.sda3.in‚0( proc.memory.availableMB‚ 1121.5078125 cpu‚25.133826756228853 network.sent‚ 5.826074e+06 disk.drbd1260.out‚ 0.39453125 disk.sda.out‚ 86.19140625 ·î7’…‚(:‡ ºèèÔ´Ê™° gpu.0.gpu‚26.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.308' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.2616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u—¹)e†‚(:Ú ºèèÔâ‚—´$ disk./.usageGB‚1172.552375793457 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 7696.32421875 network.sent‚ 5.831078e+06 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 66597.6328125 disk.dm-0.out‚ 66597.6328125) proc.memory.percent‚17.91853904724121 proc.cpu.threads‚17 disk.sda.out‚ 86.58984375# disk.nvme0c0n1.in‚ 7696.32421875" disk.nvme0n1.out‚ 66597.6328125 disk.dm-0.in‚ 7696.32421875# memory_percent‚96.5646743774414 cpu‚25.13387240202766 disk.sda3.out‚ 86.58984375 disk.dm-1.out‚ 86.58984375 disk.drbd1260.out‚ 0.3984375 disk.sda1.in‚0% proc.memory.availableMB‚ 562.84375" proc.memory.rssMB‚ 2935.7734375 network.recv‚953566( disk./.usagePercent‚78.6909737764724÷Nu2‡- _wandbruntime‚2915  _runtime‚2915wú4Hň¿ % lr-AdamW/pg2‚0.000756148994838178 trainer/global_step‚2699 % _timestamp‚1.7884908154820018e+09 _runtime‚2915.122451782 _step‚218 % lr-AdamW/pg1‚0.0007561489948381780]ÄÔÊ‰Ä _runtime‚2915.122451782 _step‚218 % lr-AdamW/pg1‚0.000756148994838178 % lr-AdamW/pg2‚0.000756148994838178 trainer/global_step‚2699 % _timestamp‚1.7884908154820018e+09Úuº\ 2Š- _wandbruntime‚2915  _runtime‚2915.€Å[ò'‹ìO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191311843800064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183727194687888 trainer/global_step‚2699 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2893967865856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚183727177648528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107045371 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2894166202880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107068388 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884908154831674e+09 _step‚219 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188539925747200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47203344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2762707994467 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚191312634447360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42110041 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212272224403456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209378280477696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _runtime‚2915.123723208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2771918052864 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47203344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2771918052864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27061497 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27061495 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59865044 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191311843800064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2772690697216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15049099 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222277120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188539925747200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47203342 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191311843800064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183727194687888 trainer/global_step‚2699 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2893967865856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚183727177648528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107045371 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2894166202880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107068388 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884908154831674e+09 _step‚219 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188539925747200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47203344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2762707994467 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚191312634447360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42110041 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212272224403456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209378280477696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _runtime‚2915.123723208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2771918052864 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47203344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2771918052864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.alløó5W.current‚555 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27061497 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27061495 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59865044 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191311843800064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2772690697216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15049099 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222277120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188539925747200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47203342 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚188607643689472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188607643689472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2894999707648 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27071145 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2895208517120 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2772922798592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42125489 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212348837337600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2772922798592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚183793244084288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42124948 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚186556970258201 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 trainer/global_step‚2699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107106999 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47220216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47220216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59886781 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212348604588032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191381339145728 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209453628820480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27071143 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15054344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183793261123648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191381339145728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191380548485120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188607625686528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _step‚220 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107106999 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59886781 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚188607643689472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188607643689472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2894999707648 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27071145 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2895208517120 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2772922798592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42125489 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212348837337600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2772922798592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚183793244084288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42124948 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚186556970258201 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 trainer/global_step‚2699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107106999 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47220216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47220216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59886781 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212348604588032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191381339145728 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209453628820480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27071143 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15054344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183793261123648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191381339145728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191380548485120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188607625686528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _step‚220 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107106999 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59886781 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15315700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚د“°Ò" 2946430265344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 % _timestamp‚1.7884908625077722e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚2749 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚194747883113984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48047042 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213145697616896 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚187030516475888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2822169753088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2812778579575 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109000990 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48047044 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60930931 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚216092336705024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191924940716544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48047042 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60953946 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42859597 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚194747092466688 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚187030499436528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚189843295055463¶ÖÖŠ÷'žñO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191924940716544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48047042 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60953946 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42859597 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚194747092466688 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚187030499436528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚189843295055463 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15315160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2822942397440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2822942397440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108977973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15315700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2946430265344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 % _timestamp‚1.7884908625077722e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚2749 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚194747883113984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48047042 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213145697616896 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚187030516475888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2822169753088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2812778579575 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109000990 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48047044 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60930931 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚216092336705024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ßë×®2Ÿ- _wandbruntime‚2963  _runtime‚2963,¯FÉü& öM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2947485592576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48063916 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2947679721984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚189909574943333 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48063916 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194129408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2823174498816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚194816587812352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109016558 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚191992640655872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚213221022019584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109039601 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218069504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216168507612160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚187096582911648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60975683 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚216168725681664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 trainer/global_step‚2749 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48063918 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42873941 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109016558 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191992622652928 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _runtime‚2963.015007166 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚187096565872288 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60952642 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191992622652928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60975683 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚213221045959680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚189910362631309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚194815797151744 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48063918 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2823947156480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 % _timestamp‚1.7884908633742173e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27553545 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚191992640655872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2823174498816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075g¸æ ¡ûM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60952642 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2823947156480 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109039601 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218069504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216168507612160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚187096582911648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60975683 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚216168725681664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 trainer/global_step‚2749 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48063918 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42873941 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_tzs¥Brain_batch_end/allocated_bytes.all.peak‚ 11892003840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109016558 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191992622652928 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _runtime‚2963.015007166 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚187096565872288 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60952642 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191992622652928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60975683 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚213221045959680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚189910362631309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚194815797151744 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48063918 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2823947156480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 % _timestamp‚1.7884908633742173e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27553545 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚191992640655872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2823174498816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2947485592576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48063916 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2947679721984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚189909574943333 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48063916 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194129408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2823174498816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚194816587812352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109016558 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚191992640655872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚213221022019584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219911845675008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 179462656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62019833 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2998754859008 % _timestamp‚1.7884909189253423e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193196687428571 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚219912049077760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48890742 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2873194097664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3018.565783534 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198182341133312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219911845675008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 179462656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 L?DeviceStatsMonitor.on_train_batch_start/activeŒ|rd .small_pool.freed‚62019833 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2998754859008 % _timestamp‚1.7884909189253423e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193196687428571 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚219912049077760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48890742 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2873194097664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3018.565783534 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198182341133312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚193263755004417 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62041544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2874198856704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15587971 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43623916 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110949160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198251836478976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚198251836478976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚195377619619328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48907616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211778048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43623362 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198251045818368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚216988463098880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62064585 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110972203 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚195377637622272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 trainer/global_step‚2799 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚195377619619328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219988462557696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 110949160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚193262967316441 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2873426199040 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110972203 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62041544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28035945 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28035943 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219988250779648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216988439158784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48907616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2874198856704 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚3019.430516834 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48907618 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187837952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚195377637622272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2999999458816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2863079656153 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62041544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2874198856704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15587971 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43623916 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110949160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198251836478976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚198251836478976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚195377619619328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48907616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211778048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43623362 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198251045818368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚216988463098880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62064585 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110972203 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚195377637622272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 trainer/global_step‚2799 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚195377619619328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219988462557696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 110949160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚193262967316441 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2873426199040 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110972203 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62041544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28035945 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28035943 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219988250779648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216988439158784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48907616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2874198856704 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚3019.430516834 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48907618 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187837952 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚195377637622272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2999999458816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2863079656153 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ä† Ð2¶- _wandbruntime‚3019  _runtime‚3019*h\·Ö 2loss/SRqqt3/assignment_loss‚1.5376499507454087 0loss/FBt4/assignment_loss‚0.2772296251067728 1loss/SRqqt1/detection_loss‚2.2310539335012436 1loss/SRqqt2/detection_loss‚2.0995004881512034 .loss/FBt2/detection_loss‚3.030233318155462 epoch‚2 trainer/global_step‚2799 _step‚229 0loss/FRt4/assignment_loss‚1.8074656314320035 1loss/FBt1/assignment_loss‚0.33786909051361197 .loss/FRt4/detection_loss‚2.002988280687067 _runtime‚ 3019.43253137 2loss/SRqqt2/assignment_loss‚1.6121668973655412 .loss/FRt3/detection_loss‚2.077773843740308 % _timestamp‚1.7884909197922816e+09 0loss/FRt2/assignment_loss‚3.0310438573360443 1loss/FBt2/assignment_loss‚0.19806178004099903 0loss/FBt3/assignment_loss‚0.6304228918937346 1loss/SRqqt3/detection_loss‚1.6117916835136101 1loss/SRqqt4/detection_loss‚1.1319032503745041 /loss/FBt1/detection_loss‚2.1922858736731787 %loss/total_loss‚41.84666186618754 1loss/SRqqt1/assignment_loss‚1.482211048818297 2loss/SRqqt4/assignment_loss‚0.8750351715983974 /loss/FBt4/detection_loss‚0.5626672316811201 0loss/FRt1/assignment_loss‚3.6331559932509134 0loss/FRt3/assignment_loss‚2.0058014503745145 /loss/FRt1/detection_loss‚2.5503425764483074 /loss/FRt2/detection_loss‚2.3059988475981235 /loss/FBt3/detection_loss‚2.6220091501871745  wa¸Û /loss/FRt1/detection_loss‚2.5503425764483074 /loss/FRt2/detection_loss‚2.3059988475981235 /loss/FBt3/detection_loss‚2.6220091501871745 2loss/SRqqt3/assignment_loss‚1.5376499507454087 0loss/FBt4/assignment_loss‚0.2772296251067728 1loss/SRqqt1/detection_loss‚2.2310539335012436 1loss/SRqqt2/detection_loss‚2.0995004881512034 .loss/FBt2/detection_loss‚3.030233318155462 epoch‚2 trainer/global_step‚2799 _step‚229 0loss/FRt4/assignment_loss‚1.8074656314320035 1loss/FBt1/assignment_loss‚0.33786909051361197 .loss/FRt4/detection_loss‚2.002988280687067 _runtime‚ 3019.43253137 2loss/SRqqt2/assignment_loss‚1.6121668973655412 .loss/FRt3/detection_loss‚2.077773843740308 % _timestamp‚1.7884909197922816e+09 0loss/FRt2/assignment_loss‚3.0310438573360443 1loss/FBt2/assignment_loss‚0.19806178004099903 0loss/FBt3/assignment_loss‚0.6304228918937346 1loss/SRqqt3/detection_loss‚1.6117916835136101 1loss/SRqqt4/detection_loss‚1.1319032503745041 /loss/FBt1/detection_loss‚2.1922858736731787 %loss/total_loss‚41.84666186618754 1loss/SRqqt1/assignment_loss‚1.482211048818297 2loss/SRqqt4/assignment_loss‚0.8750351715983974 /loss/FBt4/detection_loss‚0.5626672316811201 0loss/FRt1/assignment_loss‚3.6331559932509134 0loss/FRt3/assignment_loss‚2.0058014503745145åco ’¹‚(:‡ ²éèÔ棃­ gpu.0.gpu‚11.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.748' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cù?vº‚(:ë ²éèÔښݯ disk.dm-0.in‚ 8022.05859375$ memory_percent‚96.56572341918945) proc.memory.percent‚17.91853904724121% disk./.usageGB‚1172.8345031738281 disk.dm-1.in‚0 disk.dm-1.out‚ 90.02734375 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 8022.05859375! disk.nvme0n1.in‚ 8022.05859375& proc.memory.availableMB‚ 562.671875 proc.cpu.threads‚17 network.recv‚989403 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.4140625 disk.sda1.in‚0( disk.nvme0c0n1.out‚68754.66259765625& disk.nvme0n1.out‚68754.66259765625 cpu‚25.132720059047557 network.sent‚ 6.081001e+06) disk./.usagePercent‚78.70990758168975 disk.sda3.out‚ 90.02734375 disk.sda.in‚0 disk.sda.out‚ 90.02734375 disk.sda2.in‚0# disk.dm-0.out‚68750.91259765625" proc.memory.rssMB‚ 2935.7734375 disk.sda2.out‚0ñÒц’»‚(:‡ ÁéèÔèÖ⬠gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.281' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0562 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Kfix¼‚(:í ÁéèÔÆ–¿® disk.dm-1.out‚ 90.40234375 network.recv‚996232& disk.nvme0n1.out‚69037.58056640625 disk.drbd1260.in‚0 disk.sda.out‚ 90.40234375 disk.sda1.out‚0 cpu‚25.132583535520016 disk.sda3.in‚0 disk.drbd1260.out‚ 0.4140625$ memory_percent‚96.56422138214111) proc.memory.percent‚17.91853904724121 proc.cpu.threads‚17 network.sent‚ 6.189817e+06% disk./.usageGB‚1172.9359855651855# disk.nvme0c0n1.in‚ 8022.05859375! disk.nvme0n1.in‚ 8022.05859375 disk.sda3.out‚ 90.40234375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.out‚0) disk./.usagePercent‚78.71671815012314( disk.nvme0c0n1.out‚69037.58056640625 disk.dm-0.in‚ 8022.05859375# disk.dm-0.out‚69037.58056640625 disk.sda.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 562.91796875" proc.memory.rssMB‚ 2935.7734375ÚdIÜ’½‚(:‡ ÐéèÔéú÷± gpu.0.gpu‚47.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚130.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0604 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U­}¾‚(:ò ÐéèÔêÛ ·! disk.nvme0n1.in‚ 8022.05859375& disk.nvme0n1.out‚69547.71142578125 disk.sda.out‚ 90.83203125$ memory_percent‚96.56271934509277) proc.memory.percent‚17.91853904724121% disk./.usageGB‚1173.1870040893555# disk.nvme0c0n1.in‚ 8022.05859375 disk.dm-0.in‚ 8022.05859375 disk.drbd1260.out‚ 0.41796875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 network.sent‚ 6.197579e+06) disk./.usagePercent‚78.73356421389876 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 563.1640625 cpu‚25.13244350736815 proc.cpu.threads‚17 network.recv‚ 1.000403e+06# disk.dm-0.out‚69547.71142578125 disk.sda3.out‚ 90.83203125 disk.dm-1.in‚0 disk.dm-1.out‚ 90.83203125" proc.memory.rssMB‚ 2935.7734375( disk.nvme0c0n1.out‚69547.71142578125î²U&2¿- _wandbruntime‚3067  _runtime‚30676? ÞÆÀÀ _step‚230 & lr-AdamW/pg1‚0.0007486589997707994 & lr-AdamW/pg2‚0.0007486589997707994 trainer/global_step‚2849 $ _timestamp‚1.788490967698727e+09 _runtime‚3067.339115635`×ÄËÁÅ _runtime‚3067.339115635 _step‚230 & lr-AdamW/pg1‚0.0007486589997707994 & lr-AdamW/pg2‚0.0007486589997707994 trainer/global_step‚2849 $ _timestamp‚1.788490967698727e+09æ_Ä%2Â- _wandbruntime‚3067  _runtime‚3067IèÎïê ÃìO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49734444 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201617589799936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201618380447232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63108735 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193637160051888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201617589799936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198694934649344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44358591 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15849894 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142720 trainer/global_step‚2849 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2912919749791 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚196550079801679 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112843177 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220680507955200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198694934649344 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223732016231424 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2912149101383 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201618380447232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198694916646400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚196549292113911 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28508695 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198694916646400 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49734444 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201617589799936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201618380447232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63108735 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193637160051888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201617589799936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198694934649344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44358591 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15849894 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142720 trainer/global_step‚2849 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2912919749791 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚196550079801679 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112843177 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220680507955200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198694934649344 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223732016231424 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2912149101383 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201618380447232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198694916646400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚196549292113911 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28508695 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198694916646400 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44372978 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193703209448288 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49751318 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2913150241261 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63130446 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63153487 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220755880238080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201687085145600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49751318 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63153487 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63130446 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201686294484992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49751316 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15855176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223808191045120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 % _timestamp‚1.7884909685608327e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2923677899264 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112904805 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112904805 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _runtime‚ 3068.20201442 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198762616585728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2924450556928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚201687085145600 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193703226487648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198762634588672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49751316 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220755856297984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2923677899264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28518343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15854635 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3052533070848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198762616585728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005´ˆn±ÇúM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63130446 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201686294484992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49751316 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15855176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223808191045120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 % _timestamp‚1.7884909685608327e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2923677899264 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112904805 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112904805 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _runtime‚ 3068.20201442 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198762616585728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2924450556928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚201687085145600 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193703226487648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198762634588672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49751316 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220755856297984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2923677899264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28518343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15854635 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3052533070848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198762616585728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 g#¼‰OB9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚196617147377525 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112881762 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223808413308928 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44372978 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193703209448288 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49751318 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2913150241261 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63130446 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63153487 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220755880238080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201687085145600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49751318 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63153487 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0èT­2È- _wandbruntime‚3068  _runtime‚3068ø\}]É× epoch‚2 _step‚233 0loss/FRt3/assignment_loss‚2.3617128109445376 %loss/total_loss‚43.85179141297729 trainer/global_step‚2849 /loss/FRt1/assignment_loss‚3.521189698667237 0loss/FRt2/assignment_loss‚3.4489317764254177 1loss/FBt4/assignment_loss‚0.15789784148149538 1loss/SRqqt3/detection_loss‚1.5898337090638146 /loss/FBt1/detection_loss‚3.4976093769073486 .loss/FRt1/detection_loss‚2.790123968864932 _runtime‚ 3068.20370235 0loss/FRt4/assignment_loss‚1.1463554611011428 2loss/SRqqt4/assignment_loss‚0.9323323719700177 /loss/FBt2/detection_loss‚3.6087063044309615 /loss/FBt3/detection_loss‚2.5968513233321056 % _timestamp‚1.7884909685635252e+09 2loss/SRqqt1/assignment_loss‚1.4142984017729758 1loss/SRqqt4/detection_loss‚0.9541207159558932 2loss/SRqqt3/assignment_loss‚1.3388038636810744 0loss/FBt2/assignment_loss‚0.1797840618644841 /loss/FRt4/detection_loss‚1.7295013255610758 2loss/SRqqt2/assignment_loss‚1.8280266518222874 0loss/FBt1/assignment_loss‚0.5937402443960309 1loss/FBt3/assignment_loss‚0.17433793819093676 1loss/SRqqt2/detection_loss‚2.2315075551641397 .loss/FRt2/detection_loss‚2.474498482311526 /loss/FRt3/detection_loss‚2.4031242892450217 1loss/SRqqt1/detection_loss‚2.2803483709692953 /loss/FBt4/detection_loss‚0.5981548688535386ß6ø¡bÊÜ 2loss/SRqqt4/assignment_loss‚0.9323323719700177 /loss/FBt2/detection_loss‚3.6087063044309615 /loss/FBt3/detection_loss‚2.5968513233321056 % _timestamp‚1.7884909685635252e+09 2loss/SRqqt1/assignment_loss‚1.4142984017729758 1loss/SRqqt4/detection_loss‚0.9541207159558932 2loss/SRqqt3/assignment_loss‚1.3388038636810744 0loss/FBt2/assignment_loss‚0.1797840618644841 /loss/FRt4/detection_loss‚1.7295013255610758 2loss/SRqqt2/assignment_loss‚1.8280266518222874 0loss/FBt1/assignment_loss‚0.5937402443960309 1loss/FBt3/assignment_loss‚0.17433793819093676 1loss/SRqqt2/detection_loss‚2.2315075551641397 .loss/FRt2/detection_loss‚2.474498482311526 /loss/FRt3/detection_loss‚2.4031242892450217 1loss/SRqqt1/detection_loss‚2.2803483709692953 /loss/FBt4/detection_loss‚0.5981548688535386 epoch‚2 _step‚233 0loss/FRt3/assignment_loss‚2.3617128109445376 %loss/total_loss‚43.85179141297729 trainer/global_step‚2849 /loss/FRt1/assignment_loss‚3.521189698667237 0loss/FRt2/assignment_loss‚3.4489317764254177 1loss/FBt4/assignment_loss‚0.15789784148149538 1loss/SRqqt3/detection_loss‚1.5898337090638146 /loss/FBt1/detection_loss‚3.4976093769073486 .loss/FRt1/detection_loss‚2.790123968864932 _runtime‚ 3068.20370235 0loss/FRt4/assignment_loss‚1.1463554611011428é"ÿW"’Ë‚(:‡ ßéèÔ…å™± gpu.0.gpu‚44.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.012' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ܼPDuÌ‚(:ê ßéèÔÊ㋳( disk.nvme0c0n1.out‚70028.73095703125 disk.sda3.out‚ 91.19140625 disk.dm-1.out‚ 91.19140625 disk.drbd1260.out‚ 0.41796875 disk.sda.in‚0$ memory_percent‚96.56460285186768( proc.memory.availableMB‚ 562.85546875( disk./.usagePercent‚78.7434548089766 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚70028.73095703125# disk.dm-0.out‚70028.73095703125 proc.cpu.threads‚17 network.sent‚ 6.200765e+06 disk.nvme0c0n1.in‚ 8076.28125 disk.nvme0n1.in‚ 8076.28125 disk.dm-0.in‚ 8076.28125 disk.sda3.in‚0" proc.memory.rssMB‚ 2935.7734375) proc.memory.percent‚17.91853904724121 cpu‚25.132276786189568 network.recv‚ 1.003865e+06% disk./.usageGB‚1173.3343811035156 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 91.19140625 disk.sda1.in‚0LÌ?ªÍ‚(:’ îéèÔÕή gpu.0.gpu‚49.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.074' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.614800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ)šá`΂(:Õ îéèÔ뇰 disk.sda.in‚0) disk./.usagePercent‚78.75188001692844 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚8171.25! disk.dm-0.out‚70190.388671875$ memory_percent‚96.56469821929932 network.sent‚ 6.308389e+06 disk.dm-1.out‚ 91.5859375 disk.sda.out‚ 91.5859375 disk.sda1.out‚0( proc.memory.availableMB‚ 562.83984375" proc.memory.rssMB‚ 2935.7734375 proc.cpu.threads‚17 network.recv‚ 1.0094e+06% disk./.usageGB‚1173.4599227905273 disk.sda3.out‚ 91.5859375 disk.drbd1260.in‚0 disk.drbd1260.out‚0.421875 disk.sda2.out‚0 disk.nvme0c0n1.in‚8171.25) proc.memory.percent‚17.91853904724121 disk.sda1.in‚0& disk.nvme0c0n1.out‚70190.388671875$ disk.nvme0n1.out‚70190.388671875 disk.dm-0.in‚8171.25 cpu‚25.132095093203056–ò뤒Ï‚(:‡ ýéèÔÈ«¢¯ gpu.0.gpu‚33.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.3312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ºPùaЂ(:Ö ýéèÔÚ°Œ³ network.recv‚ 1.013984e+06& disk.nvme0c0n1.out‚70533.162109375 disk.sda3.in‚0 disk.sda3.out‚ 91.99609375 disk.drbd1260.in‚0 disk.sda2.in‚0% disk./.usageGB‚1173.6870422363281 disk.drbd1260.out‚0.421875 disk.sda.in‚0' proc.memory.availableMB‚ 562.5859375" proc.memory.rssMB‚ 2935.7734375) proc.memory.percent‚17.91853904724121 cpu‚25.131973270555) disk./.usagePercent‚78.76712219350205 disk.nvme0c0n1.in‚8171.25$ disk.nvme0n1.out‚70533.162109375! disk.dm-0.out‚70533.162109375 disk.sda.out‚ 91.99609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 network.sent‚ 6.314574e+06 disk.nvme0n1.in‚8171.25 disk.dm-0.in‚8171.25$ memory_percent‚96.56624794006348 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 91.99609375üê®Ç2Ñ- _wandbruntime‚3117  _runtime‚3117w¯&-ÇÒÁ & lr-AdamW/pg1‚0.0007460146887838437 & lr-AdamW/pg2‚0.0007460146887838437 trainer/global_step‚2899 % _timestamp‚1.7884910182703445e+09 _runtime‚3117.910881904 _step‚234U…8óÌÓÆ & lr-AdamW/pg1‚0.0007460146887838437 & lr-AdamW/pg2‚0.0007460146887838437 trainer/global_step‚2899 % _timestamp‚1.7884910182703445e+09 _runtime‚3117.910881904 _step‚234ê`s2Ô- _wandbruntime‚3117  _runtime‚3117¸Ãÿ&ò'ÕìO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2973697498112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3103954962432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚235 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2972924853760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚205053629113856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205053629113856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202079931615744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50578144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3117.912352315 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16117270 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64220652 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2962990334899 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚227551903996928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3103758722560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2899 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚205052838466560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114798796 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚227551683816960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196940464800528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205052838466560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45107824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50578144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224447925094400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196940481839888 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202079913612800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚199903472174787 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2973697498112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2962219686491 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50578142 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224447949034496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 % _timestamp‚1.7884910182713299e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64197637 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202079931615744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3103954962432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚235 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2972924853760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚205053629113856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205053629113856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202079931615744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50578144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3117.912352315 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16117270 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64220652 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2962990334899 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚227551903996928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3103758722560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2899 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚205052838466560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114798796 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚227551683816960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196940464800528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205052838466560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45107824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50578144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224447925094400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196940481839888 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202079913612800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚199903472174787 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2973697498112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2962219686491 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50578142 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224447949034496ë¤i¼¾2×- _wandbruntime‚3118  _runtime‚3118í59ØõM D>DeviceStatsMonitor.on_train_batch_end/reserved_pù÷ŽÂ&bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64219348 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3104815640064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114837407 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚202147631555072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50595018 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50595018 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50595016 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16122607 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202147613552128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205121543151616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3105007672320 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚205122333812224 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚227628089077248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192032256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2963991474985 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2974702257152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45123352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199969752062657 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114814364 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45122807 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202147613552128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2963220826369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29000745 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114814364 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215972352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2974702257152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64242389 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224523273437184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114837407 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64219348 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197006531236288§¬ 'ÙúM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3104815640064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114837407 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚202147631555072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50595018 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50595018 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50595016 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16122607 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202147613552128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205121543151616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3105007672320 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚205122333812224 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚227628089077248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192032256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2963991474985 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2974702257152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45123352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199969752062657 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114814364 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45122807 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202147613552128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2963220826369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29000745 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114814364 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215972352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2974702257152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64242389 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224523273437184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114837407 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64219348 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197006531236288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64219348 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39ì&ðÀ2Ú- _wandbruntime‚3118  _runtime‚3118ÖIZÛÔ 2loss/SRqqt4/assignment_loss‚1.0229930924103847 1loss/FBt3/assignment_loss‚0.05039656674489379 .loss/FRt1/detection_loss‚2.290073019153667 1loss/SRqqt2/detection_loss‚2.4329495346173644 /loss/FBt4/detection_loss‚0.5182200172088213 0loss/FRt1/assignment_loss‚3.2325045275238327 /loss/FRt2/detection_loss‚2.5688381177006345 0loss/SRqqt4/detection_loss‚1.044175076106238 0loss/FBt2/assignment_loss‚0.6616069625791473 0loss/SRqqt3/detection_loss‚1.811335609604915 0loss/FRt4/assignment_loss‚1.3347805041187215 2loss/SRqqt1/assignment_loss‚1.5063914282199664 _runtime‚3118.885094705 /loss/FBt2/detection_loss‚2.7788244485855103 %loss/total_loss‚43.32285823408351 $ _timestamp‚1.788491019244795e+09 /loss/FRt3/assignment_loss‚2.738676352934404 2loss/SRqqt3/assignment_loss‚1.5166193460497785 /loss/FRt3/detection_loss‚1.9880710287527605 /loss/FRt4/detection_loss‚1.5422740025480963 1loss/SRqqt1/detection_loss‚2.0631468143218603 /loss/FRt2/assignment_loss‚3.617705012812759 1loss/SRqqt2/assignment_loss‚2.587572124786675 0loss/FBt1/assignment_loss‚0.3689012086639802 1loss/FBt4/assignment_loss‚0.13744394975448937 /loss/FBt3/detection_loss‚2.5061941544214883 epoch‚2 /loss/FBt1/detection_loss‚3.0031653344631195 trainer/global_step‚2899 _step‚237Ša³Ø_ÜÙ 1loss/SRqqt1/detection_loss‚2.0631468143218603 /loss/FRt2/assignment_loss‚3.617705012812759 1loss/SRqqt2/assignment_loss‚2.587572124786675 0loss/FBt1/assignment_loss‚0.3689012086639802 1loss/FBt4/assignment_loss‚0.13744394975448937 /loss/FBt3/detection_loss‚2.5061941544214883 epoch‚2 /loss/FBt1/detection_loss‚3.0031653344631195 trainer/global_step‚2899 _step‚237 2loss/SRqqt4/assignment_loss‚1.0229930924103847 1loss/FBt3/assignment_loss‚0.05039656674489379 .loss/FRt1/detection_loss‚2.290073019153667 1loss/SRqqt2/detection_loss‚2.4329495346173644 /loss/FBt4/detection_loss‚0.5182200172088213 0loss/FRt1/assignment_loss‚3.2325045275238327 /loss/FRt2/detection_loss‚2.5688381177006345 0loss/SRqqt4/detection_loss‚1.044175076106238 0loss/FBt2/assignment_loss‚0.6616069625791473 0loss/SRqqt3/detection_loss‚1.811335609604915 0loss/FRt4/assignment_loss‚1.3347805041187215 2loss/SRqqt1/assignment_loss‚1.5063914282199664 _runtime‚3118.885094705 /loss/FBt2/detection_loss‚2.7788244485855103 %loss/total_loss‚43.32285823408351 $ _timestamp‚1.788491019244795e+09 /loss/FRt3/assignment_loss‚2.738676352934404 2loss/SRqqt3/assignment_loss‚1.5166193460497785 /loss/FRt3/detection_loss‚1.9880710287527605 /loss/FRt4/detection_loss‚1.5422740025480963í;!‚/Ý‚(:’ ŒêèÔÖ¼†® gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.433' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.886599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™Yå iÞ‚(:Þ ŒêèÔêÿ¼² disk.sda2.out‚0" proc.memory.rssMB‚ 2935.7734375) disk./.usagePercent‚78.80074161491355 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.42578125 disk.sda1.in‚0$ memory_percent‚96.56496047973633& proc.memory.availableMB‚ 562.796875) proc.memory.percent‚17.91853904724121 cpu‚25.131911259847655 network.sent‚ 6.3196e+06 disk.sda.out‚ 92.46484375 disk.sda2.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1174.1879959106445 disk.nvme0c0n1.in‚8171.25( disk.nvme0c0n1.out‚70948.34228515625# disk.dm-0.out‚70948.34228515625 disk.sda3.out‚ 92.46484375 disk.dm-1.out‚ 92.46484375 disk.sda.in‚0 disk.sda1.out‚0 network.recv‚ 1.017549e+06 disk.nvme0n1.in‚8171.25& disk.nvme0n1.out‚70948.34228515625 disk.dm-0.in‚8171.25 disk.sda3.in‚0 disk.drbd1260.in‚0è’ß‚(:‡ ›êèÔö¸Ô¬ gpu.0.gpu‚50.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.736' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7472 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00œ³ià‚(:Þ ›êèÔ›«®¯ disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚71441.34619140625 disk.nvme0n1.in‚8171.25 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.42578125 disk.dm-0.in‚8171.25# disk.dm-0.out‚71441.34619140625& proc.memory.availableMB‚ 562.546875) proc.memory.percent‚17.91853904724121 network.recv‚ 1.022953e+06 disk.sda.in‚0 disk.sda.out‚ 92.77734375 disk.nvme0c0n1.in‚8171.25& disk.nvme0n1.out‚71441.34619140625 disk.sda3.in‚0 disk.sda3.out‚ 92.77734375" proc.memory.rssMB‚ 2935.7734375 network.sent‚ 6.427255e+06# disk./.usageGB‚1174.18599319458 disk.dm-1.out‚ 92.77734375$ memory_percent‚96.56648635864258 cpu‚25.131813244724942 proc.cpu.threads‚17 disk.sda2.out‚0) disk./.usagePercent‚78.80060721095808 disk.drbd1260.in‚0d'£6á‚(:’ ªêèÔï°–® gpu.0.gpu‚37.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.818' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.763600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ØcŽdâ‚(:Ù ªêèÔí¢ßµ disk.dm-1.in‚0 disk.dm-1.out‚ 93.203125 disk.drbd1260.in‚0 disk.sda.out‚ 93.203125% disk.nvme0n1.out‚71951.5166015625 disk.dm-0.in‚8288.875) disk./.usagePercent‚78.80218216930872% disk./.usageGB‚1174.2094612121582 disk.sda3.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚8288.875" disk.dm-0.out‚71951.5166015625" proc.memory.rssMB‚ 2935.7734375) proc.memory.percent‚17.91853904724121 network.sent‚ 6.434171e+06 network.recv‚ 1.026454e+06 disk.drbd1260.out‚ 0.4296875 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚8288.875$ memory_percent‚96.56496047973633 disk.sda3.out‚ 93.203125' disk.nvme0c0n1.out‚71951.5166015625& proc.memory.availableMB‚ 562.796875 cpu‚25.131643861922985 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0µ%ȱã‚(:… ¹êèÔ¡ò³² gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.46' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃÛ½)iä‚(:Þ ¹êèÔÁ–¸ disk.sda3.out‚ 93.60546875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚8288.875( disk./.usagePercent‚78.8358717367846 disk.drbd1260.out‚ 0.4296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚8288.875% disk.nvme0n1.out‚72294.2587890625 disk.dm-0.in‚8288.875$ memory_percent‚96.56496047973633% disk./.usageGB‚1174.7114601135254' disk.nvme0c0n1.out‚72294.2587890625& proc.memory.availableMB‚ 562.796875) proc.memory.percent‚17.91853904724121 proc.cpu.threads‚17 network.recv‚ 1.031412e+06 disk.dm-1.out‚ 93.60546875 disk.sda.in‚0 disk.sda.out‚ 93.60546875" disk.dm-0.out‚72294.2587890625" proc.memory.rssMB‚ 2935.7734375 cpu‚25.131455600898857 network.sent‚ 6.438552e+06 disk.sda3.in‚0*.JÃ2å- _wandbruntime‚3167  _runtime‚3167$ý”ÇæÁ & lr-AdamW/pg1‚0.0007432974604878584 & lr-AdamW/pg2‚0.0007432974604878584 trainer/global_step‚2949 % _timestamp‚1.7884910680468829e+09 _runtime‚3167.687419206 _step‚238òBoÌçÆ & lr-AdamW/pg1‚0.0007432974604878584 & lr-AdamW/pg2‚0.0007432974604878584 trainer/global_step‚2949 % _timestamp‚1.7884910680468829e+09 _runtime‚3167.687419206 _step‚238î¨Ú½2è- _wandbruntime‚3167  _runtime‚3167˜%ZIãéëO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51421844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3023176553984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16384960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16384427 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228215342233600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228215366173696 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚208488087133184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚205464928582144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45857922 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65286539 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚203256076860127 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚200243786588528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚208488877780480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3156485635584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3156295687168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3023949198336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3013060920007 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51421844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65309554 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116731398 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29473495 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 _step‚239 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51421842 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚205464910579200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213888512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3023176553984 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65309554 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚200243803627888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116708381 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3012290271599 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚231371637920768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2949 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3023949198336 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45858457 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 % _timestamp‚1.7884910680485115e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚203256864547895 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚208488087133184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚208488087133184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚205464928582144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45857922 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65286539 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚203256076860127 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚200243786588528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚208488877780480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3156485635584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3156295687168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3023949198336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3013060920007 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51421844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65309554 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116731398 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29473495 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 _step‚239 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51421842 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚205464910579200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213888512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3023176553984 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65309554 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚200243803627888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116708381 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3012290271599 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚231371637920768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2949 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3023949198336 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45858457 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 % _timestamp‚1.7884910680485115e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚203256864547895 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚208488087133184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚205532628521472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116770009 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231448251295232 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116746966 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚205532610518528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚208556791818240 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116770009 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65331291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29483143 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚205532610518528 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3024953957376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65308250 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51438716 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚208557582478848 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3168.757531347 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3157336357888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65331291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16390303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚208557582478848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚228290714516480 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _step‚240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚205532628521472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45872899 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚203323932123741 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200309853024288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3014062060093Ÿj«íûM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3157536778752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3168.757531347 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3157336357888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65331291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16390303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚208557582478848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚228290714516480 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _step‚240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚205532628521472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45872899 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚203323932123741 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200309853024288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3014062060093 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚205532628521472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116770009 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231448251295232 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116746966 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚205532610518528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚208556791818240 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116770009 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65331291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29483143 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚205532610518528 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3024953957376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65308250 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51438716 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚208557582478848 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211923335799808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118664000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203547125415888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16652411 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66375441 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118640983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16652948 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3063131505115 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚208849907545600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46608845 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3073428254208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231982759372800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 trainer/global_step‚2999 _step‚243 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211924126447104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3208703663104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚231982783312896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29955895 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206610256921003 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235191681118720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52265544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3074200898560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66375441 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52265544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3062360856707 _runtime‚3217.784094666 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3074200898560 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211923335799808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118664000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203547125415888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16652411 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66375441 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118640983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16652948 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3063131505115 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚208849907545600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46608845 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3073428254208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231982759372800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 trainer/global_step‚2999 _step‚243 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211924126447104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3208703663104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚231982783312896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29955895 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206610256921003 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235191681118720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52265544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3074200898560óñª 2ý- _wandbruntime‚3218  _runtime‚3218ÇJÕ*0þöM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3075205657600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52282416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232058107715584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29965543 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206676536808873 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211992040484864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚208917607484928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D¸©Ì >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29965545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52282416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 _step‚244 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66397152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884911190150309e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235267868125184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3209948247552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 trainer/global_step‚2999 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66420193 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118679568 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66420193 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3074432999936 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211778048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚208917625487872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66397152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206677324496849 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203613191851648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚211992831145472 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118702611 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235268079903232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232058131655680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203613174812288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52282418 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46623373 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118702611 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118679568 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚208917625487872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚211992831145472 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52282418 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211778048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚208917625487872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66397152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206677324496849 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203613191851648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚211992831145472 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118702611 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235268079903232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232058131655680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203613174812288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52282418 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46623373 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118702611 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118679568 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚208917625487872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚211992831145472 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52282418 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29965543 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206676536808873 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211992040484864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚208917607484928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29965545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52282416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 _step‚244 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66397152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884911190150309e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235267868125184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3209948247552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 trainer/global_step‚2999ôÐðŠ52€- _wandbruntime‚3218  _runtime‚3218¹üò´WÑ 1loss/SRqqt1/assignment_loss‚1.634984083036531 1loss/FBt2/assignment_loss‚0.12216484536313349 0loss/FBt3/assignment_loss‚0.3085177465088366 /loss/FRt4/assignment_loss‚1.353737845140345 1loss/SRqqt1/detection_loss‚1.9439258149691991 1loss/SRqqt3/detection_loss‚1.5677805513362675 %loss/total_loss‚41.28845849719109 .loss/FRt3/assignment_loss‚2.81766218846699 2loss/SRqqt4/assignment_loss‚0.7082752731374719 0loss/SRqqt2/detection_loss‚2.193034462238613 /loss/FBt2/detection_loss‚2.2591853274239435 /loss/FBt3/detection_loss‚2.3922788160187856 % _timestamp‚1.7884911190177755e+09 0loss/FRt2/assignment_loss‚3.3167016235264866 /loss/FRt1/detection_loss‚2.6461063790179433 1loss/SRqqt2/assignment_loss‚2.191129202513318 1loss/SRqqt3/assignment_loss‚1.824812063961016 /loss/FRt3/detection_loss‚2.2954868876709127 .loss/FBt1/detection_loss‚2.738090753555298 trainer/global_step‚2999 0loss/FRt1/assignment_loss‚2.6438624206043424 epoch‚2 .loss/FRt2/detection_loss‚2.270398917524098 _runtime‚3218.658015685 /loss/FBt1/assignment_loss‚0.393893602969391 1loss/FBt4/assignment_loss‚0.05791427021208231 /loss/FRt4/detection_loss‚1.4738889959659063 1loss/SRqqt4/detection_loss‚1.1998405667868528 /loss/FBt4/detection_loss‚0.9347858592433234 _step‚245Êt¯l\‚Ö /loss/FBt1/assignment_loss‚0.393893602969391 1loss/FBt4/assignment_loss‚0.05791427021208231 /loss/FRt4/detection_loss‚1.4738889959659063 1loss/SRqqt4/detection_loss‚1.1998405667868528 /loss/FBt4/detection_loss‚0.9347858592433234 _step‚245 1loss/SRqqt1/assignment_loss‚1.634984083036531 1loss/FBt2/assignment_loss‚0.12216484536313349 0loss/FBt3/assignment_loss‚0.3085177465088366 /loss/FRt4/assignment_loss‚1.353737845140345 1loss/SRqqt1/detection_loss‚1.9439258149691991 1loss/SRqqt3/detection_loss‚1.5677805513362675 %loss/total_loss‚41.28845849719109 .loss/FRt3/assignment_loss‚2.81766218846699 2loss/SRqqt4/assignment_loss‚0.7082752731374719 0loss/SRqqt2/detection_loss‚2.193034462238613 /loss/FBt2/detection_loss‚2.2591853274239435 /loss/FBt3/detection_loss‚2.3922788160187856 % _timestamp‚1.7884911190177755e+09 0loss/FRt2/assignment_loss‚3.3167016235264866 /loss/FRt1/detection_loss‚2.6461063790179433 1loss/SRqqt2/assignment_loss‚2.191129202513318 1loss/SRqqt3/assignment_loss‚1.824812063961016 /loss/FRt3/detection_loss‚2.2954868876709127 .loss/FBt1/detection_loss‚2.738090753555298 trainer/global_step‚2999 0loss/FRt1/assignment_loss‚2.6438624206043424 epoch‚2 .loss/FRt2/detection_loss‚2.270398917524098 _runtime‚3218.658015685õú–ƒ‚(:’ õêèÔ¾¹æ­ gpu.0.gpu‚25.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.369' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.273799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÕÜ•p„‚(:å õêèÔÕîÛ¯ disk.sda3.out‚95.53125 disk.sda.out‚95.53125' proc.memory.availableMB‚ 562.1484375" proc.memory.rssMB‚ 2935.7734375) proc.memory.percent‚17.91853904724121 cpu‚25.13109217812668) disk./.usagePercent‚78.80567360006069 disk.sda2.in‚0! disk.nvme0n1.in‚ 8421.83203125 disk.dm-1.in‚0 disk.dm-1.out‚95.53125 disk.drbd1260.out‚0.4375$ memory_percent‚96.56891822814941% disk./.usageGB‚1174.2614860534668 disk.sda2.out‚0& disk.nvme0n1.out‚73599.81689453125# disk.dm-0.out‚73599.81689453125 disk.sda.in‚0 proc.cpu.threads‚17 network.sent‚ 6.563509e+06 network.recv‚ 1.049882e+06 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 8421.83203125( disk.nvme0c0n1.out‚73599.81689453125 disk.dm-0.in‚ 8421.83203125 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0kRêÇ…‚(:’ „ëèԞѰ­ gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚132.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.422600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝxÐñ{†‚(:ð „ëèÔéÆ€´ proc.cpu.threads‚17 network.sent‚ 6.673952e+06 network.recv‚ 1.056041e+06) disk./.usagePercent‚78.80567103998534 disk.sda3.out‚ 96.0859375 disk.drbd1260.out‚ 0.44140625& disk.nvme0n1.out‚74133.96923828125 disk.dm-0.in‚ 8421.83203125# disk.dm-0.out‚74133.96923828125% disk./.usageGB‚1174.2614479064941 disk.sda.in‚0! disk.nvme0n1.in‚ 8421.83203125# memory_percent‚96.5688943862915( proc.memory.availableMB‚ 562.15234375" proc.memory.rssMB‚ 2935.7734375) proc.memory.percent‚17.91853904724121 disk.dm-1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 8421.83203125( disk.nvme0c0n1.out‚74133.96923828125 cpu‚25.130960411116167 disk.sda3.in‚0 disk.dm-1.out‚ 96.0859375 disk.drbd1260.in‚0 disk.sda.out‚ 96.0859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0·ù·’‡‚(:‡ “ëèÔà‰¦­ gpu.0.gpu‚51.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.754' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1508 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠÐeC|ˆ‚(:ñ “ëèÔèèÁ² disk.drbd1260.in‚0 disk.sda1.in‚0$ memory_percent‚96.56894207000732( proc.memory.availableMB‚ 562.14453125( disk./.usagePercent‚78.8040960816347# disk.nvme0c0n1.in‚ 8421.83203125! disk.nvme0n1.in‚ 8421.83203125& disk.nvme0n1.out‚74362.66455078125 disk.sda3.in‚0 disk.dm-1.out‚ 96.45703125 disk.sda.in‚0 disk.sda.out‚ 96.45703125 disk.sda1.out‚0 disk.sda2.out‚0 cpu‚25.13084456251698 proc.cpu.threads‚17 network.sent‚ 6.677138e+06$ disk./.usageGB‚1174.237979888916 disk.dm-0.in‚ 8421.83203125 disk.sda3.out‚ 96.45703125 disk.drbd1260.out‚ 0.44140625 disk.sda2.in‚0" proc.memory.rssMB‚ 2935.7734375) proc.memory.percent‚17.91853904724121 network.recv‚ 1.059586e+06( disk.nvme0c0n1.out‚74362.66455078125# disk.dm-0.out‚74362.66455078125 disk.dm-1.in‚0B¢“=2‰- _wandbruntime‚3268  _runtime‚3268ØøŸ¨ÆŠÀ trainer/global_step‚3049 $ _timestamp‚1.788491168758616e+09 _runtime‚3268.398889787 _step‚246 & lr-AdamW/pg1‚0.0007376465321118649 & lr-AdamW/pg2‚0.0007376465321118649ÜܘwË‹Å & lr-AdamW/pg1‚0.0007376465321118649 & lr-AdamW/pg2‚0.0007376465321118649 trainer/global_step‚3049 $ _timestamp‚1.788491168758616e+09 _runtime‚3268.398889787 _step‚246öÞ;8â2Œ- _wandbruntime‚3268  _runtime‚3268Ï|s¶êO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67487358 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120596602 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206850430164528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189948416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 trainer/global_step‚3049 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚212234904512000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67464343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30438297 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3261134923776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47359141 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215358584466432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3123679954432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235750176512000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53109244 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215359375113728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 # _timestamp‚1.78849116875955e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67464343 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3123679954432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53109242 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120573585 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚215358584466432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53109244 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3261324872192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚212234922514944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16920846 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213888512 _runtime‚3268.399975282 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3124452598784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30438295 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3113202090223 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚.Ì<Û: 516 _step‚247 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235750200452096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3112431441815 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53109242 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚209963649294111 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516ñôt5õ'ŽïO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120596602 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206850430164528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189948416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 trainer/global_step‚3049 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚212234904512000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67464343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30438297 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3261134923776 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47359141 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215358584466432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3123679954432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235750176512000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53109244 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215359375113728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 # _timestamp‚1.78849116875955e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67464343 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3123679954432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53109242 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120573585 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚215358584466432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53109244 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3261324872192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚212234922514944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16920846 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213888512 _runtime‚3268.399975282 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3124452598784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30438295 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3113202090223 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 371748352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _step‚247 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235750200452096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3112431441815 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53109242 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚209963649294111 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67487358 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112÷zGÁº2- _wandbruntime‚3269  _runtime‚3269‚Ò¿|ü&öM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3113432581693 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212302622454272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3124684700160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53126118 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67509095 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53126116 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3262381824512 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120635213 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120612170 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3262183500800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3114203230309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3125457357824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30447945 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235825548794880 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212302604451328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215427289151488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47374724 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47374182 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239087708355584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 % _timestamp‚1.7884911695976975e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3125457357824 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210029929181981 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212302604451328 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53126116 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67509095 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 _step‚248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30447943 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚3049 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215428079812096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215428079812096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16926779 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210030716869957 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120612170 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005Ól”h€$‘ûM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212302622454272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53126116 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3262381824512 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120635213 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120612170 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3262183500800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3114203230309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3125457357824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30447945 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235825548794880 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212302604451328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215427289151488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47374724 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47374182 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239087708355584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 % _timestamp‚1.7884911695976975e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3125457357824 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210029929181981 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212302604451328 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53126116 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67509095 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 _step‚248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30447943 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚3049 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215428079812096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215428079812096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16926779 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 371748352 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210030716869957 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120612170 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3113432581693 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212302622454272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3124684700160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.læ´ßüarge_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53126118 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67509095 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30920695 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 trainer/global_step‚3099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53952944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215619919481344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48110003 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3313655177728 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚218793833133056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17189306 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215619901478400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3174704299008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68576260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213316253979451 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222277120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3313853514752 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30920697 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 3318.10684284 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3173931654656 _step‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53952942 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68553245 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚242831248828928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚242831471106048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239517617591296 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3163272675331 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68553245 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215619901478400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122506187 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3174704299008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122529204 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53952944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17188770 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213317041667219 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53952942 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239517593651200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218793833133056 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198337024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚218794623780352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884912184660854e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48109465 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3162502026923 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114¸—¼ö'¢ðO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3174704299008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68576260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213316253979451 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222277120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3313853514752 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30920697 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 3318.10684284 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3173931654656 _step‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53952942 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68553245 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚242831248828928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚242831471106048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239517617591296 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3163272675331 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68553245 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215619901478400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122506187 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3174704299008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122529204 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53952944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17188770 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213317041667219 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53952942 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239517593651200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218793833133056 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198337024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚218794623780352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884912184660854e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48109465 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3162502026923 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3173931654656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68576260 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚218794623780352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30920695 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 trainer/global_step‚3099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53952944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215619919481344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48110003 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3313655177728 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚218793833133056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17189306 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215619901478400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489û8<—‘2£- _wandbruntime‚3319  _runtime‚3319‘ · ¤öM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3174936400384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚218863328478720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884912194096358e+09 _step‚252 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48124916 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17194571 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚242907646285312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68597997 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17194029 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122567815 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122567815 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215687601417728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53969816 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 XFDeviceStatsMonitoNÙâür.on_train_batch_end/requested_bytes.small_pool.freed‚ 3163503166801 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 trainer/global_step‚3099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122544772 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚210219818388288 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48124372 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚3319.050745104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3175709058048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122544772 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30930345 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚239592941993984 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3164273815417 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3174936400384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53969818 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3314902615040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68574956 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213384109243065 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213383321555089 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68597997 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239592965934080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚242907868549120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚210219835427648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30930343 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3175709058048 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53969816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚218862537818112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215687601417728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68574956 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664ÒJhv'¥ûM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68574956 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213384109243065 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213383321555089 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68597997 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239592965934080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚242907868549120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚210219835427648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30930343 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3175709058048 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53969816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚218862537818112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215687601417728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68574956 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3174936400384 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚218863328478720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884912194096358e+09 _step‚252 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48124916 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17194571 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚242907646285312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68597997 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17194029 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122567815 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122567815 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215687601417728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53969816 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3163503166801 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 trainer/global_step‚3099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122544772 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚210219818388288 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48124372 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚3319.050745104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3175709058048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122544772 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30930345 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚239592941993984 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3164273815417 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3174936400384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53969818 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3314902615040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821ü¤¥ŸâgÅ°¿ _runtime‚3367.706945386 _step‚254 % lr-AdamW/pg1‚0.000731710924977414 % lr-AdamW/pg2‚0.000731710924977414 trainer/global_step‚3149 % _timestamp‚1.7884912680664122e+09;¤Ã[汀 trainer/global_step‚3149 % _timestamp‚1.7884912680664122e+09 _runtime‚3367.706945386 _step‚254 % lr-AdamW/pg1‚0.000731710924977414 % lr-AdamW/pg2‚0.000731710924977414þ©¹–^2²- _wandbruntime‚3367  _runtime‚3367ZÊã¢!³ìO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54796642 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222229872446976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222229081799680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222229872446976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚213457090779888 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69642147 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚216670434040327 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3212572612031 trainer/global_step‚3149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219004916447744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31403097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31403095 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚246651143678464 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3224183354880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0׆ó÷'´ñO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124461806 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54796642 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48860347 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚554 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243285034730496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54796644 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚216669646352559 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚213457073740528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17456696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3213343260439 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54796644 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3366132888064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3224183354880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54796642 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222229872446976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222229081799680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222229872446976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚213457090779888 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69642147 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚216670434040327 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3212572612031 trainer/global_step‚3149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219004916447744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31403097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31403095 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚246651143678464 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219072598384128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31412745 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69663858 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48874711 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17461968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3367172477440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3214344400525 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3213573751909 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3225188100608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚216737501616173 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 trainer/global_step‚3149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124500417 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54813518 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222298577145344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3225188100608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54813518 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226458112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚246727531610624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚213523140176288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124500417 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _step‚256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3225960758272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219072616387072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69686899 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚554 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222297786484736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202518016 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48874711 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17461968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3367172477440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3214344400525 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3213573751909 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3225188100608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚216737501616173 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 trainer/global_step‚3149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124500417 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54813518 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222298577145344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3225188100608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54813518 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226458112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚246727531610624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚213523140176288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124500417 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _step‚256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3225960758272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219072616387072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69686899 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚554 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222297786484736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202518016 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219072598384128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31412745 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0€`áÖ2¸- _wandbruntime‚3368  _runtime‚3368Uõ½y'¹Ú 1loss/SRqqt3/detection_loss‚1.8112232086559137 epoch‚2 trainer/global_step‚3149 % _timestamp‚1.7884912689336743e+09 2loss/SRqqt2/assignment_loss‚1.7298121603772691 2loss/SRqqt4/assignment_loss‚1.1760839303788633 1loss/FBt1/assignment_loss‚0.14211752319242804 1loss/FBt4/assignment_loss‚0.12743315935384072 1loss/SRqqt1/detection_loss‚2.3575929570488814 0loss/SRqqt2/detection_loss‚2.020736842699673 1loss/SRqqt4/detection_loss‚0.8796457565987288 .loss/FBt1/detection_loss‚2.589120125770569 /loss/FBt4/detection_loss‚0.8073294503159228 %loss/total_loss‚42.65843058185591 _runtime‚3368.573878502 2loss/SRqqt3/assignment_loss‚1.6585498662044604 1loss/FBt3/assignment_loss‚0.19897495657205583 .loss/FRt4/detection_loss‚1.705503300š¶VE9307913 2loss/SRqqt1/assignment_loss‚1.7763697228326303 1loss/FBt2/assignment_loss‚0.09274767281021923 /loss/FBt3/detection_loss‚2.7545626878738405 0loss/FRt1/assignment_loss‚3.1321748011821025 /loss/FRt1/detection_loss‚2.5605452394968755 /loss/FRt2/detection_loss‚2.5540089805921373 _step‚257 0loss/FRt2/assignment_loss‚2.9308681388696036 0loss/FRt3/assignment_loss‚2.7322109076711865 /loss/FBt2/detection_loss‚3.2840200662612915 0loss/FRt4/assignment_loss‚1.2762261235967596 /loss/FRt3/detection_loss‚2.3605730031927425c¤Weºß 0loss/FRt4/assignment_loss‚1.2762261235967596 /loss/FRt3/detection_loss‚2.3605730031927425 1loss/SRqqt3/detection_loss‚1.8112232086559137 epoch‚2 trainer/global_step‚3149 % _timestamp‚1.7884912689336743e+09 2loss/SRqqt2/assignment_loss‚1.7298121603772691 2loss/SRqqt4/assignment_loss‚1.1760839303788633 1loss/FBt1/assignment_loss‚0.14211752319242804 1loss/FBt4/assignment_loss‚0.12743315935384072 1loss/SRqqt1/detection_loss‚2.3575929570488814 0loss/SRqqt2/detection_loss‚2.020736842699673 1loss/SRqqt4/detection_loss‚0.8796457565987288 .loss/FBt1/detection_loss‚2.589120125770569 /loss/FBt4/detection_loss‚0.8073294503159228 %loss/total_loss‚42.65843058185591 _runtime‚3368.573878502 2loss/SRqqt3/assignment_loss‚1.6585498662044604 1loss/FBt3/assignment_loss‚0.19897495657205583 .loss/FRt4/detection_loss‚1.705503300307913 2loss/SRqqt1/assignment_loss‚1.7763697228326303 1loss/FBt2/assignment_loss‚0.09274767281021923 /loss/FBt3/detection_loss‚2.7545626878738405 0loss/FRt1/assignment_loss‚3.1321748011821025 /loss/FRt1/detection_loss‚2.5605452394968755 /loss/FRt2/detection_loss‚2.5540089805921373 _step‚257 0loss/FRt2/assignment_loss‚2.9308681388696036 0loss/FRt3/assignment_loss‚2.7322109076711865 /loss/FBt2/detection_loss‚3.2840200662612915‹Ä»‚(:’ ‹ìèÔºÓ›­ gpu.0.gpu‚40.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.129599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ9²t¼‚(:é ‹ìèÔ”ý¢° disk.dm-0.in‚ 8765.60546875( proc.memory.availableMB‚ 561.96484375 cpu‚25.12980633903202 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.45703125 disk.sda1.in‚0 disk.dm-1.out‚ 99.734375) proc.memory.percent‚17.91853904724121( disk./.usagePercent‚78.7593728454406 disk.sda.in‚0 disk.sda2.out‚0" disk.dm-0.out‚76621.9365234375 disk.sda3.in‚0 disk.sda3.out‚ 99.734375 disk.dm-1.in‚0$ memory_percent‚96.57003879547119 disk.sda2.in‚0 network.sent‚ 6.927035e+06 network.recv‚ 1.096329e+06% disk./.usageGB‚1173.5715713500977 disk.sda.out‚ 99.734375 disk.sda1.out‚0" proc.memory.rssMB‚ 2935.7734375 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 8765.60546875' disk.nvme0c0n1.out‚76621.9365234375! disk.nvme0n1.in‚ 8765.60546875% disk.nvme0n1.out‚76621.9365234375¥œ’½‚(:… šìèÔÅìÁ­ gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.84' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õD¨¿|¾‚(:ñ šìèÔÙ©¾° disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 8765.60546875! disk.nvme0n1.in‚ 8765.60546875( proc.memory.availableMB‚ 562.19140625) proc.memory.percent‚17.91853904724121 network.sent‚ 7.034678e+06 network.recv‚ 1.101924e+06 disk.dm-1.out‚ 100.1015625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.4609375 disk.sda.in‚0( disk.nvme0c0n1.out‚77088.43896484375 disk.dm-0.in‚ 8765.60546875% disk./.usageGB‚1172.3414306640625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚77088.43896484375# disk.dm-0.out‚77088.43896484375# memory_percent‚96.5686559677124 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda.out‚ 100.1015625" proc.memory.rssMB‚ 2935.7734375 cpu‚25.12967444631011) disk./.usagePercent‚78.67681707184398 disk.sda3.out‚ 100.1015625è­ŒL’¿‚(:‡ ©ìèÔ’‰•¯ gpu.0.gpu‚37.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Oäì|À‚(:ñ ©ìèÔÔŒö² disk.dm-0.in‚ 8919.23828125# proc.memory.rssMB‚ 2935.77734375 network.recv‚ 1.106589e+06' disk./.usagePercent‚78.692589440021% disk./.usageGB‚1172.5764503479004 disk.sda3.in‚0 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚77474.81005859375# disk.dm-0.out‚77474.81005859375) proc.memory.percent‚17.91856288909912 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚77474.81005859375$ memory_percent‚96.56867980957031 cpu‚25.129581923293607 proc.cpu.threads‚17 network.sent‚ 7.040919e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 100.58203125 disk.sda.out‚ 100.58203125 disk.sda1.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 8919.23828125$ proc.memory.availableMB‚562.1875 disk.sda3.out‚ 100.58203125 disk.drbd1260.out‚ 0.4609375! disk.nvme0n1.in‚ 8919.23828125¦ä‚=2Á- _wandbruntime‚3416  _runtime‚3416‘/%dÇÂÁ & lr-AdamW/pg1‚0.0007286379038518765 & lr-AdamW/pg2‚0.0007286379038518765 trainer/global_step‚3199 % _timestamp‚1.7884913165010033e+09 _runtime‚3416.141382503 _step‚258VÒ ØÌÃÆ trainer/global_step‚3199 % _timestamp‚1.7884913165010033e+09 _runtime‚3416.141382503 _step‚258 & lr-AdamW/pg1‚0.0007286379038518765 & lr-AdamW/pg2‚0.0007286379038518765‚:})â2Ä- _wandbruntime‚3416  _runtime‚3416/Uò'ÅìO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55640344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70754064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70754064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884913165022943e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250471263594496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3419036099072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250471487968768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126371391 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚222389913414144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225665121113600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚222389895411200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚220023038725667 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224374272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49611975 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3263413845547 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225664330466304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3274435055104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31885495 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55640342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31885497 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3418835664896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _step‚259 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3274435055104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55640342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17726478 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚220023826413435 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17725940 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247052451869696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216760395528528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚225665121113600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55640344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70731049 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49611435 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚222389895411200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3275207699456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚222389913414144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224374272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49611975 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3263413845547 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225664330466304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3274435055104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31885495 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55640342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31885497 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3418835664896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _step‚259 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3274435055104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55640342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17726478 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚220023826413435 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17725940 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247052451869696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216760395528528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚225665121113600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55640344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70731049 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49611435 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚222389895411200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3275207699456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚222389913414144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884913165022943e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250471263594496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3419036099072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250471487968768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126371391 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚222389913414144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225665121113600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚222389895411200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚220023038725667 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3417.104318295 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3275439800832 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126433019 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31895143 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3199 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3276212458496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70775801 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247127776272384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250547656290304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3275439800832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚538 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250547878554112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3420078341632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚225733035151360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126409976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49626375 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55657216 H@DeviceStatsMonitor.on_train_Vî5Gbatch_end/segment.large_pool.current‚516 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55657216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚222457613353472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126433019 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70775801 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3264414985633 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚225733825811968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚222457595350528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31895145 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17731232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55657218 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49626915 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17731770 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0VCxe'ÉûM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3275439800832 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126433019 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31895143 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3199 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3276212458496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70775801 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247127776272384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250547656290304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3275439800832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚538 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250547878554112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3420078341632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚225733035151360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126409976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49626375 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55657216 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55657216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚222457613353472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126433019 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70775801 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3264414985633 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚225733825811968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚222457595350528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31895145 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17731232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55657218 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49626915 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17731770 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚222457595350528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216826461964288 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚260 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55657218 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3419880017920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚220090893989281 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216826479003648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚225733825811968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3417.104318295„õœÄ2Ê- _wandbruntime‚3417  _runtime‚3417`Ò£$_ËÙ 1loss/SRqqt4/detection_loss‚1.0949664266469579 /loss/FBt2/detection_loss‚3.0351139158010483 1loss/FBt2/assignment_loss‚0.37537996300670784 /loss/FRt3/assignment_loss‚2.525337021052837 1loss/SRqqt3/detection_loss‚1.7902704076520328 1loss/FBt3/assignment_loss‚0.22543726497193953 .loss/FBt3/detection_loss‚2.714540958404541 epoch‚2 _runtime‚3417.106000546 _step‚261 0loss/FRt2/assignment_loss‚3.6259994004902087 1loss/SRqqt2/detection_loss‚2.2639322297440634 .loss/FBt1/detection_loss‚2.777340292930603 %loss/total_loss‚43.22616721041974 % _timestamp‚1.7884913174656684e+09 /loss/FRt1/detection_loss‚2.4576463110067626 2loss/SRqqt2/assignment_loss‚1.8942441033820312 2loss/SRqqt3/assignment_loss‚1.4333871128037572 2loss/SRqqt4/assignment_loss‚0.9824913182744274 1loss/FBt1/assignment_loss‚0.29005769327341113 0loss/FRt1/assignment_loss‚2.7616619074886497 /loss/FRt2/detection_loss‚2.5126562683205855 /loss/FRt3/detection_loss‚2.5422323107719422 /loss/FBt4/detection_loss‚0.6554425872214462 trainer/global_step‚3199 0loss/FBt4/assignment_loss‚0.2096375624837492 0loss/FRt4/assignment_loss‚1.3151298546791077 2loss/SRqqt1/assignment_loss‚1.5987004700459932 /loss/FRt4/detection_loss‚1.9716510988771916 1loss/SRqqt1/detection_loss‚2.1729107310897424ì€5dÌÞ _runtime‚3417.106000546 _step‚261 0loss/FRt2/assignment_loss‚3.6259994004902087 1loss/SRqqt2/detection_loss‚2.2639322297440634 .loss/FBt1/detection_loss‚2.777340292930603 %loss/total_loss‚43.22616721041974 % _timestamp‚1.7884913174656684e+09 /loss/FRt1/detection_loss‚2.4576463110067626 2loss/SRqqt2/assignment_loss‚1.8942441033820312 2loss/SRqqt3/assignment_loss‚1.4333871128037572 2loss/SRqqt4/assignment_loss‚0.9824913182744274 1loss/FBt1/assignment_loss‚0.29005769327341113 0loss/FRt1/assignment_loss‚2.7616619074886497 /loss/FRt2/detection_loss‚2.5126562683205855 /loss/FRt3/detection_loss‚2.5422323107719422 /loss/FBt4/detection_loss‚0.6554425872214462 trainer/global_step‚3199 0loss/FBt4/assignment_loss‚0.2096375624837492 0loss/FRt4/assignment_loss‚1.3151298546791077 2loss/SRqqt1/assignment_loss‚1.5987004700459932 /loss/FRt4/detection_loss‚1.9716510988771916 1loss/SRqqt1/detection_loss‚2.1729107310897424 1loss/SRqqt4/detection_loss‚1.0949664266469579 /loss/FBt2/detection_loss‚3.0351139158010483 1loss/FBt2/assignment_loss‚0.37537996300670784 /loss/FRt3/assignment_loss‚2.525337021052837 1loss/SRqqt3/detection_loss‚1.7902704076520328 1loss/FBt3/assignment_loss‚0.22543726497193953 .loss/FBt3/detection_loss‚2.714540958404541 epoch‚2…oÖ(\Í‚(:… ¸ìèÔõ‡Ð® gpu.0.gpu‚37.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðgøñy΂(:î ¸ìèÔÊõÀ¯% disk./.usageGB‚1172.5574378967285( disk.nvme0c0n1.out‚77693.70654296875 disk.dm-0.in‚ 8919.23828125# disk.dm-0.out‚77693.70654296875 disk.sda3.out‚ 100.984375 disk.dm-1.out‚ 100.984375 disk.sda.out‚ 100.984375$ memory_percent‚96.57046794891357 network.sent‚ 7.045854e+06) disk./.usagePercent‚78.69131349847035# disk.nvme0c0n1.in‚ 8919.23828125 disk.drbd1260.in‚0) proc.memory.percent‚17.91856288909912 cpu‚25.12943454752849 network.recv‚ 1.110185e+06 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 561.89453125 proc.cpu.threads‚17! disk.nvme0n1.in‚ 8919.23828125& disk.nvme0n1.out‚77693.70654296875 disk.sda3.in‚0 disk.drbd1260.out‚0.46875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2935.77734375PäÑÔ‘Ï‚(:† ÇìèÔØÉ´ gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.315' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.463 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?·}Ђ(:ò ÇìèÔœŽ¾$ memory_percent‚96.56894207000732) proc.memory.percent‚17.91856288909912 proc.cpu.threads‚17 network.sent‚ 7.153912e+06 disk.dm-1.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚77818.65185546875 cpu‚25.129307215118512% disk./.usageGB‚1172.5808715820312 disk.dm-1.out‚ 101.4453125 disk.drbd1260.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 8919.23828125 disk.dm-0.in‚ 8919.23828125# proc.memory.rssMB‚ 2935.77734375) disk./.usagePercent‚78.69288615275319 disk.drbd1260.out‚0.46875 disk.sda.out‚ 101.4453125 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 8919.23828125# disk.dm-0.out‚77818.65185546875( proc.memory.availableMB‚ 562.14453125 network.recv‚ 1.115424e+06 disk.sda3.in‚0 disk.sda3.out‚ 101.4453125 disk.sda1.out‚0( disk.nvme0c0n1.out‚77818.65185546875¡æ:œÑ‚(:‘ ÖìèÔÅéï gpu.0.gpu‚45.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.656000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ª4²yÒ‚(:î ÖìèÔÀ½¬² disk.dm-1.out‚ 101.828125 disk.sda.in‚0 disk.sda.out‚ 101.828125# disk.nvme0c0n1.in‚ 8919.23828125 disk.dm-0.in‚ 8919.23828125) disk./.usagePercent‚78.69709312456335 disk.sda3.in‚0 disk.sda3.out‚ 101.828125 disk.sda2.out‚0( proc.memory.availableMB‚ 561.88671875# proc.memory.rssMB‚ 2935.77734375) proc.memory.percent‚17.91856288909912 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚ 1.118695e+06 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.47265625 disk.sda2.in‚0% disk.nvme0n1.out‚77916.3759765625# memory_percent‚96.5705156326294 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚77916.3759765625! disk.nvme0n1.in‚ 8919.23828125" disk.dm-0.out‚77916.3759765625 cpu‚25.129142248135313 disk.sda1.in‚0 network.sent‚ 7.160744e+06% disk./.usageGB‚1172.6435585021973³Tº2Ó- _wandbruntime‚3463  _runtime‚3463ñ :¤ÆÔÀ trainer/global_step‚3249 $ _timestamp‚1.788491363746228e+09 _runtime‚3463.386566514 _step‚262 & lr-AdamW/pg1‚0.0007254955867132441 & lr-AdamW/pg2‚0.0007254955867132441”½âáËÕÅ _runtime‚3463.386566514 _step‚262 & lr-AdamW/pg1‚0.0007254955867132441 & lr-AdamW/pg2‚0.0007254955867132441 trainer/global_step‚3249 $ _timestamp‚1.788491363746228e+09†/ÍÂ2Ö- _wandbruntime‚3463  _runtime‚3463HÙ¹ò'×ìO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3313484430655 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56484042 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226471424 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3312713782247 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3471448995840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128303993 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17994805 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚223377218786543 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220063717316528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225774910380544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254291520536064 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71819951 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3325459399680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3324686755328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32367897 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225774892377600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚229099579132928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56484044 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220063734355888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56484042 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71842966 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚229100369780224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250819845068800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50362702 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚229100369780224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56484044 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3324686755328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 % _timestamp‚1.7884913637473023e+09 _runtime‚3463.387764963 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚254291294064640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202531328 trainer/global_step‚3249 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3325459399680 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229099579132928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225774910380544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17994259 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250819869008896 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _step‚263 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3471651527168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71819951 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128327010 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32367895 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚223376431098775 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Þu{>9ØñO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202531328 trainer/global_step‚3249 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3325459399680 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229099579132928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225774910380544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17994259 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250819869008896 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _step‚263 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3471651527168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71819951 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128327010 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32367895 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚223376431098775 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3313484430655 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56484042 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226471424 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3312713782247 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3471448995840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128303993 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17994805 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚223377218786543 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220063717316528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225774910380544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254291520536064 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71819951 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3325459399680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3324686755328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32367897 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225774892377600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚229099579132928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56484044 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220063734355888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56484042 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71842966 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚229100369780224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250819845068800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50362702 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚229100369780224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56484044 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3324686755328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 % _timestamp‚1.7884913637473023e+09 _runtime‚3463.387764963 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚254291294064640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0‡úR¼2Ù- _wandbruntime‚3464  _runtime‚3464„jéû&ÚõM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128342578 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 _step‚264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220129800791648 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56500918 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚225842610319872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225842610319872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220129783752288 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71841662 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225842592316928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3325691501056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229169074478592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 $ _timestamp‚1.788491364615059e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18000109 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3472697830912 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56500916 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250895193411584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56500918 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56500916 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71864703 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71841662 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32377543 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3325691501056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3313714922125 _runtime‚3464.255868012 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128365621 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚223443498674413 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚223444286362389 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128342578 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 trainer/global_step‚3249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229169074478592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50377107 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3326464158720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225842592316928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254367695015936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545°ûÐ'ÛúM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71864703 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3472501604352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128365621 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3326464158720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225842592316928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254367695015936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128342578 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 _step‚264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220129800791648 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56500918 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚225842610319872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225842610319872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220129783752288 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71841662 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225842592316928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3325691501056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229169074478592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 $ _timestamp‚1.788491364615059e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18000109 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3472697830912 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56500916 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250895193411584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56500918 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56500916 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71864703 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71841662 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32377543 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3325691501056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3313714922125 _runtime‚3464.255868012 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128365621 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚223443498674413 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚223444286362389 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128342578 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 trainer/global_step‚3249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229169074478592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50377107 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B K\bÞÜ 2loss/SRqqt1/assignment_loss‚1.9826755037700587 0loss/FBt2/assignment_loss‚0.3090482812037105 /loss/FBt3/detection_loss‚2.0566398547245908 1loss/SRqqt1/detection_loss‚2.3618016333291023 epoch‚2 1loss/FBt3/assignment_loss‚0.32576698258680364 /loss/FRt3/detection_loss‚2.4178059498469033 1loss/SRqqt2/detection_loss‚1.9507632534554664 /loss/FBt4/detection_loss‚0.6163429533551027 1loss/SRqqt3/assignment_loss‚1.510284384356483 /loss/FRt1/detection_loss‚2.5782089148248946 /loss/FRt2/detection_loss‚2.5775300823152065 /loss/FRt4/detection_loss‚1.4251403172082941 1loss/SRqqt3/detection_loss‚1.4818888856645893 0loss/SRqqt4/detection_loss‚1.090733932390336 /loss/FBt1/detection_loss‚2.2842214783032735 .loss/FBt2/detection_loss‚3.033478001753489 _step‚265 0loss/FRt3/assignment_loss‚2.3139801138923284 2loss/SRqqt4/assignment_loss‚1.0465983022641878 1loss/FBt1/assignment_loss‚0.09368121933269626 $ _timestamp‚1.788491364617213e+09 /loss/FRt2/assignment_loss‚3.047221852466464 0loss/FRt4/assignment_loss‚1.3395325397622997 2loss/SRqqt2/assignment_loss‚1.6748816223537668 %loss/total_loss‚40.23850972716002 trainer/global_step‚3249 0loss/FRt1/assignment_loss‚2.6272991503987995 1loss/FBt4/assignment_loss‚0.09298451760116981 _runtime‚3464.257413234‰ð”’ß‚(:‡ åìèÔãñŠ¯ gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.862' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7724 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0, °¯Sà‚(:í åìèÔԼư disk.sda1.out‚0' disk.nvme0c0n1.out‚78338.4267578125) proc.memory.percent‚17.91856288909912) disk./.usagePercent‚78.70127167953723 disk.sda1.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 9040.1953125# proc.memory.rssMB‚ 2935.77734375 cpu‚25.12897525871492 proc.cpu.threads‚17æ?,Ô% network.sent‚ 7.165167e+06 disk.nvme0n1.in‚ 9040.1953125% disk.nvme0n1.out‚78338.4267578125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 102.3203125 disk.sda2.in‚0 disk.dm-0.in‚ 9040.1953125" disk.dm-0.out‚78338.4267578125 disk.sda3.out‚ 102.3203125 disk.dm-1.out‚ 102.3203125 disk.drbd1260.out‚ 0.47265625 disk.sda.in‚0$ memory_percent‚96.56901359558105' proc.memory.availableMB‚ 562.1328125 network.recv‚ 1.123297e+06% disk./.usageGB‚1172.7058219909668Â|—ýá‚(:’ ôìèÔŧ³ gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚132.399' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.479799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÌ8|â‚(:ñ ôìèÔ󥸷 disk.drbd1260.out‚ 0.4765625 disk.sda2.in‚0 network.recv‚ 1.129498e+06( disk.nvme0c0n1.out‚78514.32177734375$ memory_percent‚96.57237529754639( proc.memory.availableMB‚ 561.58203125# proc.memory.rssMB‚ 2935.77734375) proc.memory.percent‚17.91856288909912 cpu‚25.128961763107252 proc.cpu.threads‚17 network.sent‚ 7.273578e+06% disk./.usageGB‚1172.8308219909668 disk.nvme0n1.in‚ 9040.1953125& disk.nvme0n1.out‚78514.32177734375 disk.dm-0.in‚ 9040.1953125 disk.dm-1.in‚0 disk.dm-1.out‚ 102.7265625 disk.sda.in‚0) disk./.usagePercent‚78.70966053441921# disk.dm-0.out‚78514.32177734375 disk.sda3.in‚0 disk.sda3.out‚ 102.7265625 disk.sda.out‚ 102.7265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 9040.1953125 disk.drbd1260.in‚0=DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32850297 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185754112 % _timestamp‚1.7884914133505273e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57327742 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32850295 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226730611159651 trainer/global_step‚3299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229159907346944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51113073 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18262776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚267 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3374938455552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72931868 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72908853 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3375711099904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229159907346944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57327742 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚223367039104528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3374938455552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3375711099904 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3363555015763 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18262234 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232534827799552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209694208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229159889344000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3524307634688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232535618446848 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232535618446848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57327744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232534827799552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3362784367355 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226729823471883 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229159889344000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258111593782784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57327744 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51112529 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0¨å÷'ìñO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72931868 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72908853 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚258111384088576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚3512.990971467 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51113073 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18262776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚267 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3374938455552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72931868 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72908853 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3375711099904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229159907346944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57327742 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚223367039104528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3374938455552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3375711099904 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3363555015763 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18262234 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232534827799552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209694208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229159889344000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3524307634688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232535618446848 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232535618446848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57327744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232534827799552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3362784367355 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226729823471883 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229159889344000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258111593782784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57327744 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51112529 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32850297 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185754112 % _timestamp‚1.7884914133505273e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57327742 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32850295 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226730611159651 trainer/global_step‚3299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229159907346944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3375943201280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258187778988032 trainer/global_step‚3299 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3376715858944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3525168437248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72930564 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51127494 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130275180 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57344616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226797678735497 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229227607286272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258187999154688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130298223 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57344618 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72953605 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229227607286272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232604323145216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 % _timestamp‚1.7884914142264328e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72953605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32859943 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223433105540288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_~IoÅ end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3364556155849 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3364556155849 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3375943201280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258187778988032 trainer/global_step‚3299 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3376715858944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3525168437248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72930564 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51127494 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130275180 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57344616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226797678735497 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229227607286272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258187999154688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130298223 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57344618 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72953605 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229227607286272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232604323145216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 % _timestamp‚1.7884914142264328e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72953605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32859943 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223433105540288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328Œƒa2ð- _wandbruntime‚3513  _runtime‚3513aL7}]ñ× epoch‚2 /loss/FRt2/detection_loss‚2.4394637674088173 0loss/FRt3/assignment_loss‚3.1286386936268906 2loss/SRqqt3/assignment_loss‚1.4768048110857326 1loss/FBt4/assignment_loss‚0.14750367276791448 .loss/FRt4/detection_loss‚1.644926599550004 trainer/global_step‚3299 _step‚269 % _timestamp‚1.7884914142287552e+09 /loss/FRt1/assignment_loss‚2.578885513909009 0loss/FRt4/assignment_loss‚1.1874653891641267 2loss/SRqqt2/assignment_loss‚1.6919494283360403 .loss/FRt3/detection_loss‚2.346250276616279 1loss/SRqqt4/detection_loss‚0.6923132595606148 /loss/FBt1/detection_loss‚2.3792736497190266 /loss/FRt2/assignment_loss‚4.043732482033807 0loss/FBt1/assignment_loss‚0.1900811486153139 1loss/FBt3/assignment_loss‚0.11260231584310532 1loss/FBt2/assignment_loss‚0.20119252425502054 /loss/FRt1/detection_loss‚2.4105079447736544 /loss/FBt2/detection_loss‚2.7384801656007767 _runtime‚3513.869034566 1loss/SRqqt1/detection_loss‚2.1380042493343354 &loss/total_loss‚41.897398330418724 0loss/SRqqt2/detection_loss‚2.213015923628936 .loss/FBt3/detection_loss‚3.015788061278207 2loss/SRqqt1/assignment_loss‚1.5394096344709396 2loss/SRqqt4/assignment_loss‚0.8839975631723064 1loss/SRqqt3/detection_loss‚1.9703490256138567 /loss/FBt4/detection_loss‚0.7267622300540098ÃÇ“bòÜ .loss/FRt4/detection_loss‚1.644926599550004 trainer/global_step‚3299 _step‚269 % _timestamp‚1.7884914142287552e+09 /loss/FRt1/assignment_loss‚2.578885513909009 0loss/FRt4/assignment_loss‚1.1874653891641267 2loss/SRqqt2/assignment_loss‚1.6919494283360403 .loss/FRt3/detection_loss‚2.346250276616279 1loss/SRqqt4/detection_loss‚0.6923132595606148 /loss/FBt1/detection_loss‚2.3792736497190266 /loss/FRt2/assignment_loss‚4.043732482033807 0loss/FBt1/assignment_loss‚0.1900811486153139 1loss/FBt3/assignment_loss‚0.11260231584310532 1loss/FBt2/assignment_loss‚0.20119252425502054 /loss/FRt1/detection_loss‚2.4105079447736544 /loss/FBt2/detection_loss‚2.7384801656007767 _runtime‚3513.869034566 1loss/SRqqt1/detection_loss‚2.1380042493343354 &loss/total_loss‚41.897398330418724 0loss/SRqqt2/detection_loss‚2.213015923628936 .loss/FBt3/detection_loss‚3.015788061278207 2loss/SRqqt1/assignment_loss‚1.5394096344709396 2loss/SRqqt4/assignment_loss‚0.8839975631723064 1loss/SRqqt3/detection_loss‚1.9703490256138567 /loss/FBt4/detection_loss‚0.7267622300540098 epoch‚2 /loss/FRt2/detection_loss‚2.4394637674088173 0loss/FRt3/assignment_loss‚3.1286386936268906 2loss/SRqqt3/assignment_loss‚1.4768048110857326 1loss/FBt4/assignment_loss‚0.14750367276791448vMÆ]’ó‚(:‡ ¡íèÔê’ý® gpu.0.gpu‚35.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚118.897' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7794 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–x˜Kjô‚(:ß ¡íèÔç숰# disk.nvme0c0n1.in‚ 9210.60546875 disk.dm-1.out‚104.125 disk.sda.in‚0# proc.memory.rssMB‚ 2935.77734375 cpu‚25.128707737583788 proc.cpu.threads‚17) disk./.usagePercent‚78.71539075105811% disk.nvme0c0n1.out‚79168.20703125 disk.dm-0.in‚ 9210.60546875 disk.dm-1.in‚0 disk.sda.out‚104.125 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0n1.out‚79168.20703125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.48046875 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.91856288909912 network.recv‚ 1.148305e+06% disk./.usageGB‚1172.9162063598633! disk.nvme0n1.in‚ 9210.60546875 disk.dm-0.out‚79168.20703125 disk.sda3.out‚104.125 disk.drbd1260.in‚0$ memory_percent‚96.57282829284668' proc.memory.availableMB‚ 561.5078125 network.sent‚ 7.293402e+06BБ õ‚(:… °íèÔéù¥¯ gpu.0.gpu‚10.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚162.52' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235970076466176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3425190155776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232544886310400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3412854952463 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203402752 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 _step‚271 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51863827 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚258354703287296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚230084003532759 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58171442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18531666 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132192214 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 179462656 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33332695 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235970867113472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232544904313344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _runtime‚3562.738919985 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74020770 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚258354679347200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚261931633576960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚230083215844991 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235970076466176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58171444 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 $ _timestamp‚1.788491463098095e+09 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3576930289664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3576750827008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74020770 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3425190155776 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226670360892528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73997755 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132169197 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235970867113472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18531132 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73997755 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58171442ì%uWÛ þðO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132169197 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235970867113472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18531132 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73997755 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58171442 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58171444 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235970076466176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3425190155776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232544886310400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3412854952463 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203402752 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 _step‚271 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51863827 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚258354703287296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚230084003532759 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58171442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18531666 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segmô went.large_pool.current‚516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132192214 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 179462656 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33332695 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235970867113472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232544904313344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _runtime‚3562.738919985 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74020770 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚258354679347200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚261931633576960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚230083215844991 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235970076466176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58171444 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 $ _timestamp‚1.788491463098095e+09 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3576930289664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3576750827008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74020770 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3425190155776 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226670360892528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73997755 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328n?J2ÿ- _wandbruntime‚3563  _runtime‚3563ö9^ü&€öM PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚232612604252672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262008046307328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3413856092341 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213875200 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132230825 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3414626740957 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚236038781151232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58188316 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132230825 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58188318 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3426967559168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132207782 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚258430027689984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚230150283420629 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚236039571811840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33342345 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3426967559168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚230151071108605 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3426194901504 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74019466 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232612586249728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18536521 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262007832432128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132207782 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚258430051630080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58188318 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51878864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226736427328288 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚236039571811840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0[F]'ûM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132230825 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3414626740957 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚236038781151232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58188316 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132230825 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58188318 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3426967559168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132207782 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚258430027689984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚230150283420629 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚236039571811840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33342345 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3426967559168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚230151071108605 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3426194901504 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74019466 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232612586249728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18536521 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262007832432128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132207782 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚258430051630080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58188318 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51878864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226736427328288 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚236039571811840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚232612604252672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262008046307328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3413856092341 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213875200 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75086657 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚239405325132800 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75086657 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229973682680528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 % _timestamp‚1.7884915124612918e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59015144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134101799 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75109672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3462925537571 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3475441856000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33815097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3463696185979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3476214500352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3476214500352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52614291 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229973699719888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚239406115780096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235929883276800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚233436608218099 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134124816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _step‚275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33815095 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3629368541696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚239406115780096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 _runtime‚3612.102112701 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239405325132800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52613740 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235929901279744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75086657 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚239405325132800 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75086657 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229973682680528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 % _timestamp‚1.7884915124612918e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59015144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134101799 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75109672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3462925537571 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3475441856000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33815097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3463696185979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3476214500352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3476214500352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52614291 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229973699719888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚239406115780096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235929883276800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚233436608218099 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134124816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _step‚275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33815095 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3629368541696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚239406115780096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 _runtime‚3612.102112701 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239405325132800 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52613740 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235929901279744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235997601219072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265827865946624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚233504463481713 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚239474820478464 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _runtime‚3612.954784279 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75131409 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59032018 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚239474029817856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _step‚276 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3476446601728 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884915133137927e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3477219259392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235997583216128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3630615246848 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59032018 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚230039766155648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194129408 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134163427 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134140384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134140384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3476446601728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235997601219072 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚265828084016128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33824745 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚262197468769280 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59032016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3463926677449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3464697326065 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59032016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52629074 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚3399 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75131409 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52628542 ZFDeviceStatsMonitor.on_train_batch_end/requested_byte¿ŒàÑ$s.large_pool.freed‚230039749116288ê ['“ûM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52628542 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚230039749116288 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235997601219072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265827865946624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚233504463481713 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚239474820478464 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _runtime‚3612.954784279 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75131409 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59032018 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚239474029817856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _step‚276 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3476446601728 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884915133137927e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3477219259392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235997583216128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3630615246848 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59032018 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚230039766155648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194129408 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134163427 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134140384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134140384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3476446601728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235997601219072 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚265828084016128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33824745 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚262197468769280 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59032016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3463926677449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3464697326065 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59032016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52629074 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚3399 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75131409”Ên»¡2”- _wandbruntime‚3612  _runtime‚3612צRˆ_•Ù 0loss/FRt4/assignment_loss‚1.9190181305533962 2loss/FBt3/assignment_loss‚0.053955384354973525 1loss/SRqqt3/detection_loss‚1.4629782995002136 /loss/FBt2/detection_loss‚2.7985024094581603 trainer/global_step‚3399 _runtime‚3612.956105338 2loss/SRqqt1/assignment_loss‚1.8360511962981785 2loss/SRqqt4/assignment_loss‚0.7456022229214961 1loss/SRqqt1/detection_loss‚2.1156662229229424 0loss/FBt4/detection_loss‚0.47638142101156217 0loss/FRt2/assignment_loss‚2.9630108250512017 /loss/FRt3/assignment_loss‚2.590468450138966 2loss/SRqqt2/assignment_loss‚1.8688527149044805 /loss/FRt2/detection_loss‚2.3435056606963487 /loss/FRt3/detection_loss‚2.4690839558218918 0loss/SRqqt2/detection_loss‚2.238075484832128 _step‚277 .loss/FBt3/detection_loss‚2.485979750752449 epoch‚2 0loss/FBt1/assignment_loss‚0.5744944398424455 0loss/FBt2/assignment_loss‚0.3003323471173644 /loss/FRt4/detection_loss‚2.1422506851193153 .loss/FBt1/detection_loss‚2.647849202156067 &loss/total_loss‚42.670180621077826 0loss/FRt1/assignment_loss‚2.9732918394239327 /loss/FRt1/detection_loss‚2.7930825207578507 2loss/SRqqt3/assignment_loss‚1.5596790094795223 1loss/FBt4/assignment_loss‚0.12411744196766392 1loss/SRqqt4/detection_loss‚1.1879510059952736 $ _timestamp‚1.788491513315908e+09祭ùd–Þ $ _timestamp‚1.788491513315908e+09 0loss/FRt4/assignment_loss‚1.9190181305533962 2loss/FBt3/assignment_loss‚0.053955384354973525 1loss/SRqqt3/detection_loss‚1.4629782995002136 /loss/FBt2/detection_loss‚2.7985024094581603 trainer/global_step‚3399 _runtime‚3612.956105338 2loss/SRqqt1/assignment_loss‚1.8360511962981785 2loss/SRqqt4/assignment_loss‚0.7456022229214961 1loss/SRqqt1/detection_loss‚2.1156662229229424 0loss/FBt4/detection_loss‚0.47638142101156217 0loss/FRt2/assignment_loss‚2.9630108250512017 /loss/FRt3/assignment_loss‚2.590468450138966 2loss/SRqqt2/assignment_loss‚1.8688527149044805 /loss/FRt2/detection_loss‚2.3435056606963487 /loss/FRt3/detection_loss‚2.4690839558218918 0loss/SRqqt2/detection_loss‚2.238075484832128 _step‚277 .loss/FBt3/detection_loss‚2.485979750752449 epoch‚2 0loss/FBt1/assignment_loss‚0.5744944398424455 0loss/FBt2/assignment_loss‚0.3003323471173644 /loss/FRt4/detection_loss‚2.1422506851193153 .loss/FBt1/detection_loss‚2.647849202156067 &loss/total_loss‚42.670180621077826 0loss/FRt1/assignment_loss‚2.9732918394239327 /loss/FRt1/detection_loss‚2.7930825207578507 2loss/SRqqt3/assignment_loss‚1.5596790094795223 1loss/FBt4/assignment_loss‚0.12411744196766392 1loss/SRqqt4/detection_loss‚1.1879510059952736•ù8æä’—‚(:‡ ûíèÔ€”Ô± gpu.0.gpu‚22.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.041' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ûÎǘ‚(:ö ûíèÔœçÙ´ network.sent‚ 7.533204e+06 network.recv‚ 1.176603e+06) disk./.usagePercent‚78.65146157363776" disk.nvme0n1.in‚17243.58203125 disk.sda2.in‚0 disk.dm-0.in‚17243.58203125 disk.sda3.out‚ 106.5859375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 106.5859375 disk.sda1.out‚0$ memory_percent‚96.57111167907715' proc.memory.availableMB‚ 561.7890625% disk./.usageGB‚1171.9636154174805$ disk.nvme0c0n1.in‚17243.58203125& disk.nvme0n1.out‚90728.39111328125 disk.drbd1260.out‚ 0.5234375 disk.sda.in‚0# proc.memory.rssMB‚ 2935.77734375) proc.memory.percent‚17.91856288909912 cpu‚25.127808162349996( disk.nvme0c0n1.out‚90728.39111328125# disk.dm-0.out‚90728.39111328125 disk.sda3.in‚0 disk.dm-1.out‚ 106.5859375 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17Ó¸»’™‚(:‡ ŠîèÔâûî¯ gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.256' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ§«°|š‚(:ñ ŠîèÔòÌŠ² disk.sda3.in‚0 disk.dm-1.in‚0 network.recv‚ 1.188173e+06) disk./.usagePercent‚78.66406021641279' disk.nvme0c0n1.out‚91175.0966796875" disk.dm-0.out‚91175.0966796875 disk.drbd1260.in‚0 disk.sda.out‚ 106.953125% disk./.usageGB‚1172.1513442993164 disk.dm-0.in‚17296.26953125 disk.sda3.out‚ 106.953125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 561.77734375 cpu‚25.12766509892799 network.sent‚ 7.644297e+06$ disk.nvme0c0n1.in‚17296.26953125" disk.nvme0n1.in‚17296.26953125% disk.nvme0n1.out‚91175.0966796875 disk.sda2.out‚0$ memory_percent‚96.57118320465088) proc.memory.percent‚17.91856288909912 proc.cpu.threads‚17 disk.dm-1.out‚ 106.953125 disk.drbd1260.out‚ 0.52734375 disk.sda2.in‚0# proc.memory.rssMB‚ 2935.77734375쑦›‚(:’ ™îèÔŠÚ® gpu.0.gpu‚17.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.324' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.864800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é7…œ‚(:ú ™îèԤŸ² disk.drbd1260.out‚ 0.52734375 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚17315.85546875 disk.dm-0.in‚17315.85546875# disk.dm-0.out‚91497.34716796875$ memory_percent‚96.57180309295654 disk.dm-1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2935.77734375 proc.cpu.threads‚17% disk./.usageGB‚1172.2593688964844 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0& disk.nvme0n1.out‚91497.34716796875( proc.memory.availableMB‚ 561.67578125) proc.memory.percent‚17.91856288909912 cpu‚25.12754922268195 network.sent‚ 7.660685e+06 network.recv‚ 1.213033e+06) disk./.usagePercent‚78.67130983776741 disk.dm-1.out‚ 107.30859375 disk.sda.out‚ 107.30859375 disk.sda1.in‚0( disk.nvme0c0n1.out‚91497.34716796875" disk.nvme0n1.in‚17315.85546875 disk.sda3.out‚ 107.30859375Èå³F‚(:’ ¨îèÔׇ¯ gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚154.427' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.885399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:|~ž‚(:ó ¨îèÔ•ø÷² disk.dm-0.in‚ 17426.046875 disk.dm-1.in‚0) proc.memory.percent‚17.91856288909912 cpu‚25.127431831235516) disk./.usagePercent‚78.67969741261173# disk.dm-0.out‚92035.30810546875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.54296875 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 107.73828125 network.sent‚ 7.66721e+06( disk.nvme0c0n1.out‚92035.30810546875 disk.nvme0n1.in‚ 17426.046875 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 107.73828125$ memory_percent‚96.57180309295654 network.recv‚ 1.218016e+06" disk.nvme0c0n1.in‚ 17426.046875 disk.sda3.out‚ 107.73828125 disk.sda1.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 561.67578125# proc.memory.rssMB‚ 2935.77734375$ disk./.usageGB‚1172.384349822998& disk.nvme0n1.out‚92035.30810546875OxÃZ2Ÿ- _wandbruntime‚3661  _runtime‚3661¹1Ç Á _runtime‚3661.203165579 _step‚278 & lr-AdamW/pg1‚0.0007122466688424625 & lr-AdamW/pg2‚0.0007122466688424625 trainer/global_step‚3449 % _timestamp‚1.7884915615628273e+09'sSÌ¡Æ & lr-AdamW/pg1‚0.0007122466688424625 & lr-AdamW/pg2‚0.0007122466688424625 trainer/global_step‚3449 % _timestamp‚1.7884915615628273e+09 _runtime‚3661.203165579 _step‚278– î>V2¢- _wandbruntime‚3661  _runtime‚3661Wx©{ò'£ìO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269571643338752 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76198574 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59858844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226471424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242840573799424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59858844 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3525693556224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242840573799424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76175559 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19067034 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚236790000591207 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269571869810176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59858842 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3526466200576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚239314898246144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239314880243200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3525693556224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3682332244480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3682129713152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76198574 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19066510 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3513766771087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚265889513625600 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚236790788278975 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59858842 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34297497 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚265889537565696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 trainer/global_step‚3449 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚239314898246144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3526466200576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚239314880243200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76175559 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242841364446720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚279 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524)ª>8¤ñO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76175559 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242841364446720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚279 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269571643338752 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76198574 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59858844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226471424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242840573799424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59858844 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3525693556224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242840573799424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76175559 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19067034 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚236790000591207 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269571869810176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59858842 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3526466200576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚239314898246144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 ZLDeviceStatsMonitor.on_train_batchNhïqÝ_start/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239314880243200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3525693556224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3682332244480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3682129713152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76198574 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19066510 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3513766771087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚265889513625600 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚236790788278975 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59858842 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34297497 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚265889537565696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 trainer/global_step‚3449 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚239314898246144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3526466200576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚239314880243200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992—€3ôz2¥- _wandbruntime‚3662  _runtime‚3662ý¡æü&¦öM N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3514767911173 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136096029 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚239382598185472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚265964861968384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136072986 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19072343 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3683372835328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76197270 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3526698301952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚236857068166845 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚233343070904288 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _step‚280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202518016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19071810 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3513997262557 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269648258743808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242910069145088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59875716 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34307145 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59875716 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 % _timestamp‚1.7884915625984573e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3662.239636141 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136096029 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76197270 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59875718 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136072986 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚239382580182528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 trainer/global_step‚3449 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3526698301952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53378953 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0¯BÚ'§ûM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3683372835328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76197270 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3526698301952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚236857068166845 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚233343070904288 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _step‚280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202518016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19071810 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3513997262557 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269648258743808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242910069145088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59875716 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34307145 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59875716 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 % _timestamp‚1.7884915625984573e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3662.239636141 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136096029 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76197270 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59875718 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136072986 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚239382580182528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 trainer/global_step‚3449 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3526698301952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53378953 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3514767911173 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136096029 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚239382598185472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚265964861968384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136072986 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19072343˜Ǹ2¨- _wandbruntime‚3662  _runtime‚3662(\°ÍX©Ò /loss/FRt2/detection_loss‚2.5914441978230194 0loss/FRt1/assignment_loss‚3.4509698106692386 0loss/FRt3/assignment_loss‚2.3005757994121976 0loss/FBt1/assignment_loss‚0.5141959548625892 1loss/SRqqt2/detection_loss‚2.5102632786288406 1loss/SRqqt3/detection_loss‚1.4686335806241806 %loss/total_loss‚43.34675409884461 _step‚281 0loss/FRt4/assignment_loss‚1.3958680203982763 2loss/SRqqt4/assignment_loss‚1.1660093049208322 /loss/FBt1/detection_loss‚3.0727253556251526 2loss/SRqqt1/assignment_loss‚1.5661734112251837 2loss/SRqqt2/assignment_loss‚1.7176694172349842 .loss/FRt1/detection_loss‚2.131944317657214 .loss/FRt3/detection_loss‚2.150618238857499 .loss/FRt4/detection_loss‚2.043535488879397 1loss/SRqqt1/detection_loss‚1.9916208008160958 1loss/SRqqt4/detection_loss‚1.0866339613000553 /loss/FBt4/detection_loss‚0.5810595385622807 0loss/FBt3/assignment_loss‚0.2954708427860169 1loss/FBt4/assignment_loss‚0.05278665975590494 _runtime‚3662.240947032 /loss/FRt2/assignment_loss‚4.480034323299632 -loss/FBt3/detection_loss‚2.40046489238739 epoch‚2 trainer/global_step‚3449 % _timestamp‚1.7884915626005006e+09 0loss/SRqqt3/assignment_loss‚1.23896027696045 0loss/FBt2/assignment_loss‚0.4179165646589051 /loss/FBt2/detection_loss‚2.7211800614992776Ë(E7]ª× /loss/FRt2/assignment_loss‚4.480034323299632 -loss/FBt3/detection_loss‚2.40046489238739 epoch‚2 trainer/global_step‚3449 % _timestamp‚1.7884915626005006e+09 0loss/SRqqt3/assignment_loss‚1.23896027696045 0loss/FBt2/assignment_loss‚0.4179165646589051 /loss/FBt2/detection_loss‚2.7211800614992776 /loss/FRt2/detection_loss‚2.5914441978230194 0loss/FRt1/assignment_loss‚3.4509698106692386 0loss/FRt3/assignment_loss‚2.3005757994121976 0loss/FBt1/assignment_loss‚0.5141959548625892 1loss/SRqqt2/detection_loss‚2.5102632786288406 1loss/SRqqt3/detection_loss‚1.4686335806241806 %loss/total_loss‚43.34675409884461 _step‚281 0loss/FRt4/assignment_loss‚1.3958680203982763 2loss/SRqqt4/assignment_loss‚1.1660093049208322 /loss/FBt1/detection_loss‚3.0727253556251526 2loss/SRqqt1/assignment_loss‚1.5661734112251837 2loss/SRqqt2/assignment_loss‚1.7176694172349842 .loss/FRt1/detection_loss‚2.131944317657214 .loss/FRt3/detection_loss‚2.150618238857499 .loss/FRt4/detection_loss‚2.043535488879397 1loss/SRqqt1/detection_loss‚1.9916208008160958 1loss/SRqqt4/detection_loss‚1.0866339613000553 /loss/FBt4/detection_loss‚0.5810595385622807 0loss/FBt3/assignment_loss‚0.2954708427860169 1loss/FBt4/assignment_loss‚0.05278665975590494 _runtime‚3662.240947032™^HЫ‚(:’ ·îèÔ‡„¥¶ gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.062200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cWß´¬‚(:ô ·îèÔõÀз disk.sda.out‚ 108.05859375 disk.sda2.out‚0' proc.memory.availableMB‚ 636.0234375 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 17426.046875( disk.nvme0c0n1.out‚92407.25732421875# disk.dm-0.out‚92407.25732421875$ memory_percent‚96.11802101135254" proc.memory.rssMB‚ 2861.5546875* proc.memory.percent‚17.465543746948242 network.recv‚ 1.221317e+06) disk./.usagePercent‚78.69808131364552 disk.sda3.in‚0 disk.dm-1.out‚ 108.05859375 disk.drbd1260.out‚ 0.54296875 disk.sda1.in‚0% disk./.usageGB‚1172.6582832336426 disk.nvme0n1.in‚ 17426.046875& disk.nvme0n1.out‚92407.25732421875 disk.dm-0.in‚ 17426.046875 cpu‚25.127389818525202 proc.cpu.threads‚17 network.sent‚ 7.672307e+06 disk.sda3.out‚ 108.05859375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0X<ò@­‚(:’ ÆîèÔôŽ€´ gpu.0.gpu‚49.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.057' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.611399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 óWëq®‚(:æ ÆîèÔò͇¾$ memory_percent‚96.11804485321045" proc.memory.rssMB‚ 2861.5546875 cpu‚25.127457729961307" disk.nvme0c0n1.in‚ 17426.046875$ disk.nvme0n1.out‚92556.955078125 disk.dm-0.in‚ 17426.046875! disk.dm-0.out‚92556.955078125 disk.sda2.out‚0( proc.memory.availableMB‚ 636.01953125 proc.cpu.threads‚17) disk./.usagePercent‚78.69801295963387 disk.nvme0n1.in‚ 17426.046875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 network.recv‚ 1.227052e+06% disk./.usageGB‚1172.6572647094727 disk.sda3.out‚ 108.640625 disk.dm-1.out‚ 108.640625 disk.drbd1260.out‚0.546875 disk.sda.out‚ 108.640625* proc.memory.percent‚17.465543746948242 network.sent‚ 7.77976e+06& disk.nvme0c0n1.out‚92556.955078125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0–>UM¯‚(:’ ÕîèÔÌëί gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.897' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.379399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o.ë°‚(:ç ÕîèÔ¥“È° disk.sda3.in‚0 network.sent‚ 7.790331e+06& disk.nvme0c0n1.out‚92852.634765625 disk.dm-1.in‚0 disk.dm-1.out‚ 109.0234375 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 109.0234375" proc.memory.rssMB‚ 2861.5546875 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚92852.634765625 disk.drbd1260.out‚0.546875* proc.memory.percent‚17.465543746948242 cpu‚25.12733598231942 proc.cpu.threads‚17 netwúÿyK‡ork.recv‚ 1.23743e+06 disk.nvme0n1.in‚ 17426.046875 disk.sda3.out‚ 109.0234375$ memory_percent‚96.11656665802002( proc.memory.availableMB‚ 636.26171875( disk./.usagePercent‚78.6867673166773% disk./.usageGB‚1172.4896965026855 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 17426.046875 disk.dm-0.in‚ 17426.046875! disk.dm-0.out‚92852.634765625ºP4¦2±- _wandbruntime‚3710  _runtime‚3710‰\ DzÁ % _timestamp‚1.7884916112870917e+09 _runtime‚3710.927416541 _step‚282 & lr-AdamW/pg1‚0.0007087679784359192 & lr-AdamW/pg2‚0.0007087679784359192 trainer/global_step‚3499¤F¢Ì³Æ trainer/global_step‚3499 % _timestamp‚1.7884916112870917e+09 _runtime‚3710.927416541 _step‚282 & lr-AdamW/pg1‚0.0007087679784359192 & lr-AdamW/pg2‚0.0007087679784359192š&ÉŸy2´- _wandbruntime‚3710  _runtime‚3710cäUò'µìO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77264461 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60702542 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚273391672687616 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60702542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246276613113344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269656930764800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137967003 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246276613113344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222277120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240144180652083 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77287476 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _runtime‚3710.928574512 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚242699895212544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3499 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246275822466048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3576717900800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137990020 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240143392964315 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77264461 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19334338 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3734741922816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3576717900800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚269656954704896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚236580326256528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚236580343295888 _step‚283 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242699877209600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246275822466048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34779895 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19334886 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3575945256448 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246275822466048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3576717900800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137990020 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240143392964315 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77264461 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19334338 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3734741922816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3576717900800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚269656954704896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚236580326256528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚236580343295888 _step‚283 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242699877209600 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246275822466048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34779895 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19334886 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3575945256448 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60702542 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚273391672687616 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60702542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246276613113344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269656930764800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137967003 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246276613113344›‚µf!2·- _wandbruntime‚3711  _runtime‚3711“…![ü&¸öM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54129286 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34789543 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246345317811712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242767577148928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚284 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77309213 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138005588 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚269732303047680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246345317811712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242767595151872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19339743 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3564067847665 % _timestamp‚1.7884916121594605e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138028631 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60719418 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3577722659840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60719418 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240210460539953 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77286172 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242767577148928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246344527151104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3576950002176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60719416 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚269732279107584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77309213 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚236646409731648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77286172 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚273468092929024 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3735997465088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242767595151872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3577722659840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34789545 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183643648 _runtime‚3711.800815861 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3576950002176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3735813821440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207583744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3499 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138028631 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54129828 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240211248227929 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requeste<]ˆÜ!d_bytes.all.peak‚ 11518882821 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚236646409731648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77286172 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚273468092929024 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3735997465088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242767595151872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3577722659840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34789545 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183643648 _runtime‚3711.800815861 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3576950002176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3735813821440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207583744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3499 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138028631 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54129828 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240211248227929 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54129286 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34789543 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246345317811712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242767577148928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚284 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77309213 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138005588 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27845 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚269732303047680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246345317811712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242767595151872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19339743 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3564067847665 % _timestamp‚1.7884916121594605e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138028631 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60719418 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3577722659840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634821 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60719418 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240210460539953 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77286172 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242767577148928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246344527151104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3576950002176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60719416 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0œA*_2º- _wandbruntime‚3711  _runtime‚3711l ¼ý_»Ù /loss/FRt3/detection_loss‚2.0723261492592946 .loss/FBt3/detection_loss‚3.185874581336975 _runtime‚3711.801618864 2loss/SRqqt1/assignment_loss‚1.7991529313080452 1loss/FBt3/assignment_loss‚0.07873839833773673 .loss/FBt2/detection_loss‚3.322028309106827 trainer/global_step‚3499 _step‚285 /loss/FRt4/detection_loss‚1.6283381959354435 0loss/SRqqt1/detection_loss‚2.334909332149169 1loss/FBt4/assignment_loss‚0.17348619996407405 /loss/FRt2/detection_loss‚2.6763710686678275 0loss/FRt2/assignment_loss‚2.7364794449372725 1loss/FBt2/assignment_loss‚0.06420821019491996 /loss/FRt3/assignment_loss‚2.585857405549004 % _timestamp‚1.7884916121612318e+09 1loss/SRqqt4/detection_loss‚0.9766122573962459 /loss/FBt1/detection_loss‚2.5265664756298065 0loss/FRt4/assignment_loss‚1.4480241316336173 2loss/SRqqt2/assignment_loss‚1.5918313188212259 1loss/SRqqt2/detection_loss‚2.1569021361214773 1loss/SRqqt3/detection_loss‚1.4965852044763104 /loss/FBt4/detection_loss‚0.5100602779012902 epoch‚2 %loss/total_loss‚41.39195871101987 0loss/FRt1/assignment_loss‚2.9445519127496858 2loss/SRqqt3/assignment_loss‚1.5662997441456443 2loss/SRqqt4/assignment_loss‚0.8902881493090422 1loss/FBt1/assignment_loss‚0.29492670530453324 /loss/FRt1/detection_loss‚2.3315401707844035­'æd¼Þ /loss/FRt4/detection_loss‚1.6283381959354435 0loss/SRqqt1/detection_loss‚2.334909332149169 1loss/FBt4/assignment_loss‚0.17348619996407405 /loss/FRt2/detection_loss‚2.6763710686678275 0loss/FRt2/assignment_loss‚2.7364794449372725 1loss/FBt2/assignment_loss‚0.06420821019491996 /loss/FRt3/assignment_loss‚2.585857405549004 % _timestamp‚1.7884916121612318e+09 1loss/SRqqt4/detection_loss‚0.9766122573962459 /loss/FBt1/detection_loss‚2.5265664756298065 0loss/FRt4/assignment_loss‚1.4480241316336173 2loss/SRqqt2/assignment_loss‚1.5918313188212259 1loss/SRqqt2/detection_loss‚2.1569021361214773 1loss/SRqqt3/detection_loss‚1.4965852044763104 /loss/FBt4/detection_loss‚0.5100602779012902 epoch‚2 %loss/total_loss‚41.39195871101987 0loss/FRt1/assignment_loss‚2.9445519127496858 2loss/SRqqt3/assignment_loss‚1.5662997441456443 2loss/SRqqt4/assignment_loss‚0.8902881493090422 1loss/FBt1/assignment_loss‚0.29492670530453324 /loss/FRt1/detection_loss‚2.3315401707844035 /loss/FRt3/detection_loss‚2.0723261492592946 .loss/FBt3/detection_loss‚3.185874581336975 _runtime‚3711.801618864 2loss/SRqqt1/assignment_loss‚1.7991529313080452 1loss/FBt3/assignment_loss‚0.07873839833773673 .loss/FBt2/detection_loss‚3.322028309106827 trainer/global_step‚3499 _step‚285” Ò½‚(:’ äîèÔú„Я gpu.0.gpu‚24.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚159.589' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.917800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä*ÀÛ}¾‚(:ò äîèÔɨ¬±* proc.memory.percent‚17.465543746948242 cpu‚25.127085685694013 disk.nvme0n1.in‚ 17565.578125 network.recv‚ 1.261528e+06 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 17565.578125 disk.dm-0.in‚ 17565.578125# disk.dm-0.out‚93393.68017578125 disk.sda3.in‚0 disk.dm-1.out‚ 109.51171875 disk.sda.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚93393.68017578125 disk.sda3.out‚ 109.51171875 disk.drbd1260.out‚ 0.55078125# memory_percent‚96.1186408996582" proc.memory.rssMB‚ 2861.5546875 proc.cpu.threads‚17 network.sent‚ 7.804372e+06% disk./.usageGB‚1172.6013221740723 disk.sda.out‚ 109.51171875 disk.sda1.in‚0( disk.nvme0c0n1.out‚93393.68017578125 disk.dm-1.in‚0) disk./.usagePercent‚78.69425860914424 disk.sda1.out‚0 disk.drbd1260.in‚0& proc.memory.availableMB‚ 635.921875k'{¯¿‚(:’ óîèÔÌ„â­ gpu.0.gpu‚34.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.428' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.685599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kg#À‚(:ô óîèÔ¯ƒ¤®# disk.dm-0.out‚93892.56298828125 disk.dm-1.in‚0 disk.dm-1.out‚ 109.88671875$ memory_percent‚96.12231254577637% disk./.usageGB‚1172.6167335510254 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2861.5546875 disk.drbd1260.out‚ 0.55078125" disk.nvme0c0n1.in‚ 17565.578125 disk.nvme0n1.in‚ 17565.578125' proc.memory.availableMB‚ 635.3203125* proc.memory.percent‚17.465543746948242 cpu‚25.126955658197897 proc.cpu.threads‚17 disk.sda2.out‚0 network.recv‚ 1.267456e+06) disk./.usagePercent‚78.69529287958258 disk.sda.in‚0 disk.sda.out‚ 109.88671875 disk.dm-0.in‚ 17565.578125 disk.sda3.in‚0 disk.sda3.out‚ 109.88671875 network.sent‚ 7.912344e+06 disk.sda1.out‚0( disk.nvme0c0n1.out‚93892.56298828125& disk.nvme0n1.out‚93892.56298828125T!E°’Á‚(:‡ ‚ïèÔتå­ gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.614' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„iÅ|‚(:ñ ‚ïèÔ¤åÿ° disk.sda3.out‚ 110.28515625 disk.dm-1.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚94355.4189453125% disk.nvme0n1.out‚94355.4189453125 disk.dm-0.in‚ 17565.578125* proc.memory.percent‚17.465543746948242 disk.sda3.in‚0 disk.drbd1260.out‚ 0.5546875 disk.sda.out‚ 110.28515625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 17565.578125 disk.nvme0n1.in‚ 17565.578125$ memory_percent‚96.12233638763428" proc.memory.rssMB‚ 2861.5546875 network.sent‚ 7.917281e+06 disk.dm-1.out‚ 110.28515625 disk.drbd1260.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 635.31640625 cpu‚25.126812994188015 network.recv‚ 1.270988e+06% disk./.usageGB‚1172.6397132873535 disk.sda.in‚0 disk.sda1.out‚0" disk.dm-0.out‚94355.4189453125 proc.cpu.threads‚17) disk./.usagePercent‚78.69683506896884c î*2Ã- _wandbruntime‚3761  _runtime‚3761Vª3ÅÄ¿ & lr-AdamW/pg2‚0.0007052241328789563 trainer/global_step‚3549 # _timestamp‚1.78849166213031e+09 _runtime‚3761.770795406 _step‚286 & lr-AdamW/pg1‚0.0007052241328789563#¢LÊÅÄ & lr-AdamW/pg2‚0.0007052241328789563 trainer/global_step‚3549 # _timestamp‚1.78849166213031e+09 _runtime‚3761.770795406 _step‚286 & lr-AdamW/pg1‚0.0007052241328789563žÿ“Eõ2Æ- _wandbruntime‚3761  _runtime‚3761vékêò'ÇìO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78353363 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3626969601024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚246084892178944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884916621316164e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239883665083888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139899605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3626196956672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚246084874176000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78376378 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚273424347904000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚239883648044528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61546242 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚249711071132672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚249711861779968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249711071132672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3787307546624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35262295 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54865893 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3626969601024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243497573025191 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243496785337423 _runtime‚3761.772158324 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19603596 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 _step‚287 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚562 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139922622 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78353363 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19603034 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35262297 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125„\0–ÈñO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139899605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3626196956672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚246084874176000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78376378 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚273424347904000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚239883648044528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61546242 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚249711071132672 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚249711861779968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249711071132672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3787307546624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35262295 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54865893 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3626969601024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243497573025191 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243496785337423 _runtime‚3761.772158324 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19603596 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 _step‚287 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚562 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139922622 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78353363 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19603034 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35262297 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78353363 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3626969601024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚246084892178944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.cuõ ©êarrent‚17039360 % _timestamp‚1.7884916621316164e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239883665083888Ÿj#hx2É- _wandbruntime‚3762  _runtime‚3762Íì_ü&ÊöM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3627201702400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚239949731519648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218069504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35271943 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884916630593154e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61563118 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61563116 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19608441 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚277288055963648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139961233 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚246152592118272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚273499696246784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚288 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78375074 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3627201702400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243564640601037 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78398115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19608991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78375074 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3788359716864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194129408 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139961233 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139938190 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚246152592118272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61563116 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3614909081389 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246152574115328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54880384 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚239949714480288 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61563118 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78398115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3788553846272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 _runtime‚3762.700262255 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚246152574115328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚277288274033152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚273499720186880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35271945 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2ƒrÄ 'ËûM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139938190 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚246152592118272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61563116 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3614909081389 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772657664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246152574115328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54880384 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚239949714480288 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28114 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61563118 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78398115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3788553846272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 _runtime‚3762.700262255 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚246152574115328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚277288274033152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28114 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚273499720186880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35271945 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3627201702400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚239949731519648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218069504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35271943 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884916630593154e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61563118 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61563116 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19608441 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚277288055963648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139961233 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚246152592118272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚273499696246784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚288 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78375074 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3627201702400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243564640601037 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78398115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19608991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78375074 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3788359716864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23041 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790660608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194129408 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139961233 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 öºÃ§2Ì- _wandbruntime‚3762  _runtime‚3762N¸\_ÍÙ 2loss/SRqqt1/assignment_loss‚1.8914769752060665 /loss/FRt4/detection_loss‚1.4331662003593197 1loss/SRqqt2/detection_loss‚3.1377035812898115 % _timestamp‚1.7884916630617447e+09 _step‚289 /loss/FRt1/assignment_loss‚3.083625218142634 0loss/FRt3/assignment_loss‚2.3017198377185397 /loss/FRt2/detection_loss‚2.5012774893219016 /loss/FBt3/detection_loss‚3.4556054983820235 0loss/SRqqt1/detection_loss‚2.326988740002408 /loss/FBt2/detection_loss‚2.8773756623268127 1loss/FBt2/assignment_loss‚0.36032923623997704 1loss/FBt4/assignment_loss‚0.10975880485202479 1loss/SRqqt3/detection_loss‚1.3637209661965128 trainer/global_step‚3549 0loss/FRt2/assignment_loss‚3.1559263910566058 0loss/FRt4/assignment_loss‚1.3109376142764915 1loss/FBt1/assignment_loss‚0.38077391259970944 /loss/FRt1/detection_loss‚2.4938052312835404 /loss/FRt3/detection_loss‚1.9806025708439174 .loss/FBt1/detection_loss‚2.142744081360953 %loss/total_loss‚42.10232773546238 0loss/FBt3/assignment_loss‚0.1941559458417552 /loss/FBt4/detection_loss‚0.7238440524220844 epoch‚2 2loss/SRqqt2/assignment_loss‚1.3282085738398812 2loss/SRqqt3/assignment_loss‚1.5499508434411469 2loss/SRqqt4/assignment_loss‚0.7959038390303558 1loss/SRqqt4/detection_loss‚1.2027264694279094 _runtime‚3762.702029475ò*ŸdÎÞ 0loss/FRt2/assignment_loss‚3.1559263910566058 0loss/FRt4/assignment_loss‚1.3109376142764915 1loss/FBt1/assignment_loss‚0.38077391259970944 /loss/FRt1/detection_loss‚2.4938052312835404 /loss/FRt3/detection_loss‚1.9806025708439174 .loss/FBt1/detection_loss‚2.142744081360953 %loss/total_loss‚42.10232773546238 0loss/FBt3/assignment_loss‚0.1941559458417552 /loss/FBt4/detection_loss‚0.7238440524220844 epoch‚2 2loss/SRqqt2/assignment_loss‚1.3282085738398812 2loss/SRqqt3/assignment_loss‚1.5499508434411469 2loss/SRqqt4/assignment_loss‚0.7959038390303558 1loss/SRqqt4/detection_loss‚1.2027264694279094 _runtime‚3762.702029475 2loss/SRqqt1/assignment_loss‚1.8914769752060665 /loss/FRt4/detection_loss‚1.4331662003593197 1loss/SRqqt2/detection_loss‚3.1377035812898115 % _timestamp‚1.7884916630617447e+09 _step‚289 /loss/FRt1/assignment_loss‚3.083625218142634 0loss/FRt3/assignment_loss‚2.3017198377185397 /loss/FRt2/detection_loss‚2.5012774893219016 /loss/FBt3/detection_loss‚3.4556054983820235 0loss/SRqqt1/detection_loss‚2.326988740002408 /loss/FBt2/detection_loss‚2.8773756623268127 1loss/FBt2/assignment_loss‚0.36032923623997704 1loss/FBt4/assignment_loss‚0.10975880485202479 1loss/SRqqt3/detection_loss‚1.3637209661965128 trainer/global_step‚3549¡µv/qÏ‚(:æ ‘ïèÔ‘ïì® disk.drbd1260.in‚0 disk.sda.out‚ 110.796875 disk.nvme0n1.in‚ 17565.578125% disk.nvme0n1.out‚94450.7939453125 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 17565.578125 disk.dm-0.in‚ 17565.578125 disk.sda3.out‚ 110.796875$ memory_percent‚96.12102508544922" proc.memory.rssMB‚ 2861.5546875 disk.drbd1260.out‚ 0.5546875 disk.sda.in‚0 disk.sda1.in‚0" disk.dm-0.out‚94450.7939453125 disk.sda3.in‚0* proc.memory.percent‚17.465543746948242 proc.cpu.threads‚17) disk./.usagePercent‚78.69684914938321 disk.sda1.out‚0' disk.nvme0c0n1.out‚94450.7939453125% proc.memory.availableMB‚ 635.53125 cpu‚25.12667306260092 network.sent‚ 7.923487e+06 network.recv‚ 1.27584e+06% disk./.usageGB‚1172.6399230957031 disk.dm-1.out‚ 110.796875ß7‘Ђ(:† ‘ïèÔÕ¡¡° gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.193' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òtJ’Ñ‚(:‡  ïèÔŽ¿‘® gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gØúê†Ò‚(:û  ïèÔ“Ì•¯ disk.sda.out‚ 111.16796875 disk.sda1.in‚0$ disk.nvme0c0n1.in‚17737.09765625( disk.nvme0c0n1.out‚94548.67724609375" disk.nvme0n1.in‚17737.09765625# disk.dm-0.out‚94548.67724609375 disk.sda3.out‚ 111.16796875 network.recv‚ 1.282624e+06% disk./.usageGB‚1172.5238571166992 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.55859375( proc.memory.availableMB‚ 635.27734375) disk./.usagePercent‚78.68905986414643 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚94548.67724609375 disk.dm-0.in‚17737.09765625 disk.dm-1.out‚ 111.16796875$ memory_percent‚96.12257480621338" proc.memory.rssMB‚ 2861.5546875 disk.dm-1.in‚0* proc.memory.percent‚17.465543746948242 cpu‚25.126564566812817 proc.cpu.threads‚17 network.sent‚ 8.032143e+06-‘‡’Ó‚(:‡ ¯ïèÔΊ® gpu.0.gpu‚49.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.623' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞJj‚Ô‚(:÷ ¯ïèÔº“° disk.sda.out‚ 111.5703125 disk.sda1.out‚0& disk.nvme0n1.out‚95092.75537109375 disk.sda3.out‚ 111.5703125 disk.dm-1.out‚ 111.5703125$ memory_percent‚96.12107276916504 network.recv‚ 1.286355e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.55859375" proc.memory.rssMB‚ 2861.5546875* proc.memory.percent‚17.465543746948242 disk.sda1.in‚0$ disk.nvme0c0n1.in‚17737.09765625( disk.nvme0c0n1.out‚95092.75537109375# disk.dm-0.out‚95092.75537109375 disk.sda3.in‚0 disk.dm-1.in‚0 cpu‚25.126513040736825 network.sent‚ 8.037123e+06% disk./.usageGB‚1172.6100463867188 disk.sda.in‚0" disk.nvme0n1.in‚17737.09765625 disk.dm-0.in‚17737.09765625' proc.memory.availableMB‚ 635.5234375 proc.cpu.threads‚17) disk./.usagePercent‚78.69484409837504¢Ç‰Ó}Õ‚(:ò ¾ïèÔÝ˱ disk.sda3.in‚0 disk.sda3.out‚ 112.28515625 disk.dm-1.out‚ 112.28515625 disk.sda.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚95390.6240234375* proc.memory.percent‚17.465543746948242 proc.cpu.threads‚17 network.sent‚ 8.042278e+06 disk.drbd1260.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2861.5546875 network.recv‚ 1.28976e+06( disk./.usagePercent‚78.7017222527975% disk./.usageGB‚1172.7125358581543 disk.dm-1.in‚0 disk.drbd1260.out‚0.5625 disk.sda2.out‚0' disk.nvme0c0n1.out‚95390.6240234375" disk.nvme0n1.in‚17737.09765625 disk.sda.out‚ 112.28515625 disk.sda1.out‚0$ disk.nvme0c0n1.in‚17737.09765625 disk.dm-0.in‚17737.09765625" disk.dm-0.out‚95390.6240234375$ memory_percent‚96.12452983856201( proc.memory.availableMB‚ 634.95703125 cpu‚25.126391796377376 NœÖ‚(:‘ ¾ïèÔ´ê’¹ gpu.0.gpu‚48.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.694' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚33.93879999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s/Â2×- _wandbruntime‚3811  _runtime‚3811tÊ* ÅØ¿ % lr-AdamW/pg1‚0.000701615870703676 % lr-AdamW/pg2‚0.000701615870703676 trainer/global_step‚3599 % _timestamp‚1.7884917115782814e+09 _runtime‚3811.218742131 _step‚290/K‰–ÊÙÄ % lr-AdamW/pg1‚0.000701615870703676 % lr-AdamW/pg2‚0.000701615870703676 trainer/global_step‚3599 % _timestamp‚1.7884917115782814e+09 _runtime‚3811.218742131 _step‚290¢ñÛŽm2Ú- _wandbruntime‚3811  _runtime‚3811(ä†uµÛìO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892003840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚511 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_trainò•Ž=&_batch_start/segment.small_pool.allocated‚489 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3676448656896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249469889145344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249469871142400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55616966 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277191765043200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚253147110446592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253146319799296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281031542021632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281031747521536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884917115798593e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79442265 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19872269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3839776978432 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62389944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62389942 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79442265 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141855224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243186986871888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79465280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚509 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3677221301248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3663207878003 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3676448656896 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3811.220474865 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79465280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3677221301248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62389942 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253146319799296 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205499904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 trainer/global_step‚3599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141832207 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249469889145344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35744697 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35744695 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19871760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3663978526411 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249469871142400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62389944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55616455 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3839958538240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243186969832528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253146319799296 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648408 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205499904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 trainer/global_step‚3599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141832207 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249469889145344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28114 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35744697 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35744695 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19871760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3663978526411 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892003840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249469871142400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62389944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55616455 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3839958538240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243186969832528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634821 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3676448656896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249469889145344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249469871142400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55616966 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277191765043200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚253147110446592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253146319799296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281031542021632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644352 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281031747521536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884917115798593e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28114 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79442265 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19872269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3839776978432 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882821 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62389944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62389942 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79442265 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141855224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243186986871888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 294056960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27845 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79465280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚509 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23017 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23015 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3677221301248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23015 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27845 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3663207878003 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3676448656896 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3811.220474865 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79465280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3677221301248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664£zDA2Ý- _wandbruntime‚3812  _runtime‚3812‡ôºnü&ÞöM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246917245286169 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 % _timestamp‚1.7884917125502408e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892003840 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3677453402624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772657664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79487017 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55631997 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55631471 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19877652 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281107945318912 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79463976 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62406818 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277267137326080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3678226060288 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249537589084672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3840831932928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19877128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277267113385984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23043 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79463976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192032256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215972352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62406816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141893835 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3841023965184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3664979666497 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249537571081728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62406816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚292 trainer/global_step‚3599 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249537571081728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3678226060288 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55631471 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19877652 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281107945318912 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79463976 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892003840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62406818 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23041 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277267137326080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3678226060288 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249537589084672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3840831932928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19877128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277267113385984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B8DeviceStatsMonitor.on_train_ba“–C³†tch_end/active.all.current‚23043 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79463976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192032256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215972352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62406816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141893835 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3841023965184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 294056960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3664979666497 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249537571081728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62406816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚292 trainer/global_step‚3599 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249537571081728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3678226060288 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882821 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790660608 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 F¹M2à- _wandbruntime‚3812  _runtime‚3812Ž ¡ÃXáÒ _runtime‚3812.193366393 2loss/SRqqt3/assignment_loss‚1.4782531869287292 1loss/FBt2/assignment_loss‚0.30340244292429736 .loss/FRt2/detection_loss‚3.401949870586395 .loss/FRt3/detection_loss‚2.538683348188275 $ _timestamp‚1.788491712553107e+09 /loss/FRt1/assignment_loss‚2.812645557764414 2loss/SRqqt2/assignment_loss‚1.5535638288727829 2loss/SRqqt4/assignment_loss‚0.9138088674437612 1loss/FBt4/assignment_loss‚0.19219742916075366 /loss/FRt4/detection_loss‚1.4705458591550082 0loss/FRt2/assignment_loss‚2.4470569372177122 1loss/SRqqt4/detection_loss‚1.0553155241355503 trainer/global_step‚3599 .loss/FRt1/detection_loss‚2.371299605917286 1loss/SRqqt2/detection_loss‚2.3202878083501544 /loss/FBt3/detection_loss‚2.6401628732681273 /loss/FBt1/assignment_loss‚0.422058287425898 0loss/FBt3/assignment_loss‚0.3768159163059257 0loss/SRqqt3/detection_loss‚1.533931749376158 /loss/FBt4/detection_loss‚0.5300643387970261 %loss/total_loss‚41.19243498066042 epoch‚2 2loss/SRqqt1/assignment_loss‚1.3513305351520195 .loss/FBt1/detection_loss‚2.626077244679133 .loss/FBt2/detection_loss‚2.905737508427013 _step‚293 0loss/FRt3/assignment_loss‚2.6580919874341866 0loss/FRt4/assignment_loss‚1.4693317615379722 1loss/SRqqt1/detection_loss‚1.8198225116118407Ù¾]â× /loss/FBt1/assignment_loss‚0.422058287425898 0loss/FBt3/assignment_loss‚0.3768159163059257 0loss/SRqqt3/detection_loss‚1.533931749376158 /loss/FBt4/detection_loss‚0.5300643387970261 %loss/total_loss‚41.19243498066042 epoch‚2 2loss/SRqqt1/assignment_loss‚1.3513305351520195 .loss/FBt1/detection_loss‚2.626077244679133 .loss/FBt2/detection_loss‚2.905737508427013 _step‚293 0loss/FRt3/assignment_loss‚2.6580919874341866 0loss/FRt4/assignment_loss‚1.4693317615379722 1loss/SRqqt1/detection_loss‚1.8198225116118407 _runtime‚3812.193366393 2loss/SRqqt3/assignment_loss‚1.4782531869287292 1loss/FBt2/assignment_loss‚0.30340244292429736 .loss/FRt2/detection_loss‚3.401949870586395 .loss/FRt3/detection_loss‚2.538683348188275 $ _timestamp‚1.788491712553107e+09 /loss/FRt1/assignment_loss‚2.812645557764414 2loss/SRqqt2/assignment_loss‚1.5535638288727829 2loss/SRqqt4/assignment_loss‚0.9138088674437612 1loss/FBt4/assignment_loss‚0.19219742916075366 /loss/FRt4/detection_loss‚1.4705458591550082 0loss/FRt2/assignment_loss‚2.4470569372177122 1loss/SRqqt4/detection_loss‚1.0553155241355503 trainer/global_step‚3599 .loss/FRt1/detection_loss‚2.371299605917286 1loss/SRqqt2/detection_loss‚2.3202878083501544 /loss/FBt3/detection_loss‚2.6401628732681273¥`Ö’ã‚(:‡ ÍïèÔÓÿà® gpu.0.gpu‚24.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.114' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÍOä‚(:ô ÍïèÔ·èõ¯ disk.drbd1260.in‚0 disk.sda.out‚ 112.67578125 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 112.67578125 disk.dm-1.out‚ 112.67578125 disk.drbd1260.out‚0.5625$ disk.nvme0c0n1.in‚17737.09765625" disk.nvme0n1.in‚17737.09765625" disk.dm-0.out‚95485.9248046875" proc.memory.rssMB‚ 2861.5546875 network.recv‚ 1.295147e+06) disk./.usagePercent‚78.71047054225755 disk.dm-1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚95485.9248046875 disk.dm-0.in‚17737.09765625( proc.memory.availableMB‚ 634.94140625* proc.memory.percent‚17.465543746948242 network.sent‚ 8.048579e+06 disk.sda.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚95485.9248046875$ memory_percent‚96.12462520599365 cpu‚25.126312361242082% disk./.usageGB‚1172.8428916931152®ö›å‚(: ÜïèÔž£Ü­ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.772000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bSŠæ‚(:ö ÜïèÔù†€°$ disk.nvme0c0n1.in‚17885.00390625" disk.nvme0n1.in‚17885.00390625 disk.sda3.in‚0 disk.dm-1.out‚ 113.1015625 network.recv‚ 1.30164e+06 disk.sda2.in‚0 disk.dm-0.in‚17885.00390625# disk.dm-0.out‚95875.60107421875 disk.sda3.out‚ 113.1015625# proc.memory.rssMB‚ 2935.69140625 proc.cpu.threads‚17 network.sent‚ 8.157048e+06 disk.sda.out‚ 113.1015625& disk.nvme0n1.out‚95875.60107421875 disk.dm-1.in‚0$ memory_percent‚96.58236503601074' proc.memory.availableMB‚ 559.9453125* proc.memory.percent‚17.918038368225098 cpu‚25.127741522117166 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚95875.60107421875) disk./.usagePercent‚78.71043367717262$ disk./.usageGB‚1172.842342376709 disk.drbd1260.out‚ 0.56640625 disk.sda2.out‚0:‡½xç‚(:„ ëïèÔ¶Ôù® gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”u9óè‚(:ö ëïèÔ­¢î¯ disk.sda1.out‚0# proc.memory.rssMB‚ 2936.12109375 network.sent‚ 8.164081e+06 disk.sda3.in‚0 disk.sda.out‚ 113.546875 disk.sda1.in‚0$ disk.nvme0c0n1.in‚17885.00390625 network.recv‚ 1.305242e+06% disk./.usageGB‚1173.0300483703613 disk.dm-0.in‚17885.00390625# disk.dm-0.out‚96140.26513671875 disk.dm-1.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚17885.00390625( proc.memory.availableMB‚ 560.64453125 disk.sda.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚96140.26513671875& disk.nvme0n1.out‚96140.26513671875$ memory_percent‚96.57809734344482* proc.memory.percent‚17.920660972595215 cpu‚25.148220864404717 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.56640625) disk./.usagePercent‚78.72303078390244 disk.sda3.out‚ 113.546875 disk.dm-1.out‚ 113.546875K˜é‚(:„ úïèÔò˜Ü­ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cú\˜…ê‚(:ú úïèÔΪ± disk.sda3.out‚ 114.01171875 disk.dm-1.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚96632.51416015625 disk.dm-0.in‚17885.00390625( proc.memory.availableMB‚ 560.14453125* proc.memory.percent‚17.920660972595215) disk./.usagePercent‚78.74247839624101 disk.sda3.in‚0 disk.dm-1.out‚ 114.01171875 disk.drbd1260.out‚ 0.5703125 disk.sda1.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2936.12109375 cpu‚25.16789052429814 network.sent‚ 8.167283e+06 network.recv‚ 1.308529e+06 disk.sda.in‚0$ disk.nvme0c0n1.in‚17885.00390625( disk.nvme0c0n1.out‚96632.51416015625" disk.nvme0n1.in‚17885.00390625% disk./.usageGB‚1173.3198318481445 disk.drbd1260.in‚0 disk.sda.out‚ 114.01171875 disk.sda1.out‚0# disk.dm-0.out‚96632.51416015625$ memory_percent‚96.58114910125732 proc.cpu.threads‚17™€Qr„ë‚(:ù ‰ðèÔ¥úΰ disk.dm-0.in‚17885.00390625# disk.dm-0.out‚97047.32666015625# proc.memory.rssMB‚ 2936.12109375 cpu‚25.18687291560883 proc.cpu.threads‚17) disk./.usagePercent‚78.75077150430518 disk.dm-1.out‚ 114.44921875( disk.nvme0c0n1.out‚97047.32666015625" disk.nvme0n1.in‚17885.00390625' proc.memory.availableMB‚ 560.3984375* proc.memory.percent‚17.920660972595215 network.recv‚ 1.312963e+06 disk.sda3.in‚0 disk.sda3.out‚ 114.44921875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.5703125 disk.sda2.in‚0 disk.sda2.out‚0% disk./.usageGB‚1173.4434051513672 disk.sda.in‚0 disk.sda.out‚ 114.44921875 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚17885.00390625& disk.nvme0n1.out‚97047.32666015625$ memory_percent‚96.57959938049316 network.sent‚ 8.175353e+06g}=ì‚(:„ ‰ðèÔø鎰 gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÃ>…2í- _wandbruntime‚3897  _runtime‚3897±'QºY*îÓT ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18214 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚80461853 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚285496021718016 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚257320968018944 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚80480067 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚36405861 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚36405859 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 3714726233427 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578072500 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884917979950933e+09 _step‚294 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1005 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 143978111 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27845 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚253593401405952 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 143978111 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚80461853 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚198 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 163219456 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 580043776 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 143959895 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1019674112 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 373079040 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3892410738688 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 598046720 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 10929229312 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 580043776 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 139279360 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882821 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3125 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 11892003840 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 3728146656768 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111860 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚56680555 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚516 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3727566612992 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚63498044 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚285495858498560 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚281603447759872 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚80480067 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18214 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28114 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚20274892 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚56680753 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 3714148160927 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚257320968018944 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚247109343237872 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 143959895 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚20274696 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚257321566065664 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 3727566612992 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 3892550018048 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚253593401405952 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚253593419408896 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18216 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10558147584 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚39 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚250824069471299 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚247109326198512 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884917979950933e+09 _step‚294 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1005 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 143978111 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27845 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚253593401405952 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 143978111 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚80461853 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚198 [MDeviceStatsMonitor.on_validation_batch_start/inacMí:1tive_split_bytes.all.current‚ 163219456 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 580043776 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 143959895 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1019674112 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 373079040 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3892410738688 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 598046720 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 10929229312 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 580043776 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 139279360 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882821 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3125 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 11892003840 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 3728146656768 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111860 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚56680555 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚516 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3727566612992 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚63498044 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚285495858498560 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚281603447759872 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚80480067 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18214 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28114 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚20274892 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0¦†í2ð- _wandbruntime‚3898  _runtime‚3898qiÞ0c)ñÝR dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚281650425905152 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚247151622038608 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884917990969656e+09 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚253638869405184 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 580043776 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚63511014 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚250866233360499 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12155092992 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚516 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1046 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 3714611321891 _runtime‚3898.738107758 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚285543464611840 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚20279793 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 3893014766592 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 11892003840 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 3715189394391 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚56693632 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚285543301392384 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18216 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚250866828472359 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚516 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚80470147 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27845 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚36413841 trainer/global_step‚3626 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28114 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580043776 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 598046720 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚80470147 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚257366900766720 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚253638887408128 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚56693831 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 373079040 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 598046720 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚63511016 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3125 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚39 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1005 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1005 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18216 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 3728611405312 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 3728031361536 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚36413839 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18214 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 10929229312 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚253638869405184 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 143981161 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28114 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚253638887408128 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚257367498813440 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚80488361 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚80488361 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚257366900766720 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595111860 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1019674112 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1075 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚197 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 3892875487232 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚20279990 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚247151639077968 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1017634821 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚63511014 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 163219456[¯Ó„h)òâR NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚20279793 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 3893014766592 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 11892003840 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 3715189394391 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚56693632 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚285543301392384 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18216 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚250866828472359 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚516 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚80470147 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27845 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚36413841 trainer/global_step‚3626 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28114 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580043776 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 598046720 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚80470147 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚257366900766720 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚253638887408128 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚56693831 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 373079040 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 598046720 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚63511016 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3125 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚39 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1005 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1005 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18216 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 3728611405312 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 3728031361536 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚36413839 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18214 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 10929229312 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚253638869405184 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 143981161 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28114 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚253638887408128 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚257367498813440 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚80488361 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚80488361 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚257366900766720 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595111860 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1019674112 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1075 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚197 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 3892875487232 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019674112 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚20279990 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚247151639077968 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1017634821 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚63511014 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 163219456 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚281650425905152 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚247151622038608 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884917990969656e+09 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚253638869405184 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 580043776 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚63511014 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚250866233360499 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12155092992 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚516 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1046 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 3714611321891 _runtime‚3898.738107758 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚285543464611840§_ög„ó‚(:’ ˜ðèÔÊèì® gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚175.771' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.154199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00 '™}ô‚(:ò ˜ðèÔâÍç° disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.57421875 disk.sda.in‚0 cpu‚25.206076810912357 network.sent‚ 8.179338e+06 disk.sda1.in‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚97487.228515625! disk.dm-0.out‚97487.228515625 disk.dm-1.out‚ 114.86328125 proc.cpu.threads‚17$ disk./.usageGB‚1173.718002319336 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 18010.2578125 disk.dm-0.in‚ 18010.2578125 disk.sda3.out‚ 114.86328125 disk.sda.out‚ 114.86328125# proc.memory.rssMB‚ 2936.12109375* proc.memory.percent‚17.920660972595215) disk./.usagePercent‚78.76919995064993! disk.nvme0n1.in‚ 18010.2578125$ disk.nvme0n1.out‚97487.228515625 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚96.58129215240479( proc.memory.availableMB‚ 560.12109375 network.recv‚ 1.317446e+06Èãq2õ- _wandbruntime‚3914  _runtime‚3914›€F:Nöƒ¦ 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.24173954129219055 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3631816506385803 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01192484237253666 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12818636000156403 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 trainer/global_step‚3626 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5496863126754761 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5157784819602966 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121212482452393 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 jet/accuracy_6_of_6‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,particle/sensitivity‚0.31847429275512695 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 particle/accuracy_6_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.15887530148029327 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.35723593831062317 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.38310810923576355 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.3017241656780243 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 +jet/accuracy_2_of_2‚0.10316728055477142 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3608570694923401 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.32535654306411743 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3511326014995575 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5288633704185486 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.41603344678878784 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3432748317718506 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38415178656578064 particle/accuracy_2_of_2‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9754207134246826 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5817720293998718 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 jet/accuracy_2_of_7‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 jet/accuracy_1_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3518965542316437 A&Purity/1F¬ùRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8105269074440002 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 jet/accuracy_5_of_7‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877830743789673 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03438638523221016 6 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.284474641084671 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.041725192219018936 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17903095483779907 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.30948275327682495 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3392544686794281 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5650816559791565 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22376544773578644 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4699074327945709 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 _runtime‚3914.686361053 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.18520034849643707 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015579749830067158 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_4_of_6‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2460516095161438 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25608494877815247 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2600663900375366 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 *jet/accuracy_2_of_5‚0.4609639346599579 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47926926612854004 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.30029329657554626 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04987684637308121 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2413385808467865 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5347222089767456 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2847222089767456 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.30351966619491577 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.244047611951828 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4280303120613098 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3663793206214905 jet/accuracy_7_of_7‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.34158986806869507 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.30571630597114563 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_3_of_5‚0.0881592407822609 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34771642088890076 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49905163049697876 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2845345139503479 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31388890743255615 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2318965494632721 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.29828721284866333 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9810611605644226 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.35812580585479736 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.2348484992980957 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27485811710357666 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32432296872138977 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 6 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.699999988079071 +jet/accuracy_2_of_6‚0.49358975887298584 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 6validation_average_jet_accuracy‚0.3955293297767639 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.32716047763824463 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2749342918395996 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.30237823724746704 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3481324017047882 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 epoch‚2 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9884097576141357 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166667461395264 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3175342082977295 particle/accuracy_2_of_3‚1 particle/accuracy_5_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6454592347145081 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.24475888907909393 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3653557598590851 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49734947085380554 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27879253029823303 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 -jet/accuracy_5_of_5‚0.0026881720405071974 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9890207052230835 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02062668465077877 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_2_of_3‚0.24286548793315887 "particle/accuracy‚ 0.8388671875 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 (particle/f_score‚0.38152822852134705 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2839947044849396 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3676043152809143 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.14287087321281433 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11096202582120895 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952420592308044 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16597338020801544 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3888888657093048 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.24923744797706604 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.029742704704403877 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.36563295125961304 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32106977701187134 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.39929142594337463 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.29531601071357727 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5821894407272339 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2997075915336609 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35984310507774353 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.29136401414871216 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 % _timestamp‚1.7884918150439968e+09 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4166666567325592 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1405663937330246 *jet/accuracy_1_of_2‚0.7063074111938477 +particle/specificity‚0.9351643323898315 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5262820720672607 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.27920717000961304 jet/accuracy_3_of_7‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19419467449188232 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3846973478794098 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9825338125228882 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8959552645683289 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754093170166 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9859980940818787 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31500738859176636 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.30515140295028687 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30732816457748413 +jet/accuracy_3_of_3‚0.03042411245405674 particle/accuracy_3_of_4‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10789874196052551 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4047996699810028 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_3_of_7‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3330332636833191 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2786460816860199 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4682260751724243 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5134870409965515 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37986111640930176 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5466269254684448 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.38724517822265625 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8361958265304565 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.36927667260169983 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31470686197280884 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.02813333459198475 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.20222222805023193 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.27819743752479553 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4000000059604645 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.335416704416275 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5478000044822693 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5493530631065369 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012193029746413231 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22268250584602356 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.994087278842926 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5860118865966797 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4217739701271057 particle/accuracy_2_of_4‚1 jet/accuracy_5_of_6‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.031132077798247337 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9727310538291931 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.3307470679283142 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2619994580745697 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.47189921140670776 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2931365966796875 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5038594007492065 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992598831653595 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 jet/accuracy_1_of_7‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9895125031471252 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.012544803321361542 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25269556045532227 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.046810582280159 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9850271344184875 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.4080808162689209 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.31172841787338257 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20752665400505066 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.37848320603370667 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963724613189697 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4165150225162506 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.001855287584476173 particle/accuracy_1_of_4‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01446305401623249 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3888888657093048 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3028053045272827 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.013926682062447071 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973641633987427 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.380405992269516 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2541394531726837 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.012022480368614197 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.990384578704834 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 particle/accuracy_7_of_7‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.37070873379707336 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9918761849403381 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9914723634719849 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2718396484851837 particle/accuracy_4_of_5‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.23603633046150208 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1226029098033905 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2568306028842926 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_1_of_7‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2413385808467865 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41712918877601624 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25967252254486084 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29670700430870056 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3583333492279053 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.3020086884498596 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_7‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.29828721284866333 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.27447718381881714 *jet/accuracy_1_of_5‚0.9342997670173645 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.28787875175476074 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.942708432674408 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 _step‚296 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07499999552965164 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9936344623565674 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5085354447364807 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.026341648772358894 particle/accuracy_5_of_7‚1 *"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_4‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2935185134410858 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚42.42638609272915 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9858543872833252 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3297223746776581 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30513688921928406 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926124215126038 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 6 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.699999988079071 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9932798743247986 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3086341917514801 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.30753669142723083 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9670635461807251 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3472222089767456 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166567325592 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9278959631919861 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 jet/accuracy_6_of_7‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6985048651695251 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.028697839006781578 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4123295545578003 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3146190047264099 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 +jet/accuracy_3_of_6‚0.11538461595773697 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5583397746086121 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.34505894780158997 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5015804767608643 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42283764481544495 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3315083086490631 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06870070099830627 particle/accuracy_5_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5275022983551025 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4934210479259491 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.000310019851895049‚¬sÃ2 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0912698432803154 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3157914876937866 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 Zí/~÷ˆ¦ ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_7‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.29828721284866333 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.27447718381881714 *jet/accuracy_1_of_5‚0.9342997670173645 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.28787875175476074 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.942708432674408 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 _step‚296 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07499999552965164 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9936344623565674 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5085354447364807 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.026341648772358894 particle/accuracy_5_of_7‚1 *"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_4‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2935185134410858 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚42.42638609272915 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9858543872833252 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3297223746776581 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30513688921928406 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926124215126038 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 6 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.699999988079071 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9932798743247986 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3086341917514801 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.30753669142723083 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9670635461807251 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3472222089767456 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166567325592 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9278959631919861 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 jet/accuracy_6_of_7‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6985048651695251 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.028697839006781578 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4123295545578003 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3146190047264099 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 +jet/accuracy_3_of_6‚0.11538461595773697 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5583397746086121 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.34505894780158997 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5015804767608643 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42283764481544495 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3315083086490631 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06870070099830627 particle/accuracy_5_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5275022983551025 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4934210479259491 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0912698432803154 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3157914876937866 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.24173954129219055 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3631816506385803 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01192484237253666 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12818636000156403 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 trainer/global_step‚3626 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5496863126754761 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5157784819602966 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121212482452393 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 jet/accuracy_6_of_6‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,particle/sensitivity‚0.31847429275512695 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 particle/accuracy_6_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.15887530148029327 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.35723593831062317 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.38310810923576355 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.3017241656780243 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 +jet/accuracy_2_of_2‚0.10316728055477142 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3608570694923401 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.32535654306411743 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3511326014995575 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5288633704185486 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.41603344678878784 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3432748317718506 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38415178656578064 particle/accuracy_2_of_2‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9754207134246826 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5817720293998718 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 jet/accuracy_2_of_7‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 jet/accuracy_1_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3518965542316437 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8105269074440002 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 jet/accuracy_5_of_7‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877830743789673 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03438638523221016 6 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.284474641084671 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.041725192219018936 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17903095483779907 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.30948275327682495 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3392544686794281 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5650816559791565 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22376544773578644 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4699074327945709 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 _runtime‚3914.686361053 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.18520034849643707 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015579749830067158 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_4_of_6‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2460516095161438 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25608494877815247 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2600663900375366 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 *jet/accuracy_2_of_5‚0.4609639346599579 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47926926612854004 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.30029329657554626 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04987684637308121 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2413385808467865 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5347222089767456 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2847222089767456 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.30351966619491577 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.244047611951828 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4280303120613098 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3663793206214905 jet/accuracy_7_of_7‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.34158986806869507 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.30571630597114563 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_3_of_5‚0.0881592407822609 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34771642088890076 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49905163049697876 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2845345139503479 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31388890743255615 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2318965494632721 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.29828721284866333 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9810611605644226 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.35812580585479736 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.2348484992980957 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27485811710357666 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32432296872138977 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 6 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.699999988079071 +jet/accuracy_2_of_6‚0.49358975887298584 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 6validation_average_jet_accuracy‚0.3955293297767639 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.32716047763824463 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2749342918395996 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.30237823724746704 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3481324017047882 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 epoch‚2 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9884097576141357 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166667461395264 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3175342082977295 particle/accuracy_2_of_3‚1 particle/accuracy_5_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6454592347145081 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.24475888907909393 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3653557598590851 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49734947085380554 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27879253029823303 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 -jet/accuracy_5_of_5‚0.0026881720405071974 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9890207052230835 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02062668465077877 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_2_of_3‚0.24286548793315887 "particle/accuracy‚ 0.8388671875 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 (particle/f_score‚0.38152822852134705 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2839947044849396 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3676043152809143 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.14287087321281433 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11096202582120895 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952420592308044 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16597338020801544 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3888888657093048 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.24923744797706604 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.029742704704403877 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.36563295125961304 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32106977701187134 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.39929142594337463 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.29531601071357727 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5821894407272339 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2997075915336609 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35984310507774353 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.29136401414871216 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 % _timestamp‚1.7884918150439968e+09 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4166666567325592 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1405663937330246 *jet/accuracy_1_of_2‚0.7063074111938477 +particle/specificity‚0.9351643323898315 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5262820720672607 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.27920717000961304 jet/accuracy_3_of_7‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19419467449188232 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3846973478794098 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9825338125228882 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8959552645683289 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754093170166 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9859980940818787 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31500738859176636 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.30515140295028687 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30732816457748413 +jet/accuracy_3_of_3‚0.03042411245405674 particle/accuracy_3_of_4‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10789874196052551 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4047996699810028 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_3_of_7‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3330332636833191 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2786460816860199 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4682260751724243 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5134870409965515 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37986111640930176 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5466269254684448 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.38724517822265625 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8361958265304565 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.36927667260169983 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31470686197280884 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.02813333459198475 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.20222222805023193 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.27819743752479553 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4000000059604645 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.335416704416275 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5478000044822693 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5493530631065369 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012193029746413231 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22268250584602356 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.994087278842926 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5860118865966797 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4217739701271057 particle/accuracy_2_of_4‚1 jet/accuracy_5_of_6‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.031132077798247337 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9727310538291931 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.3307470679283142 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2619994580745697 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.47189921140670776 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2931365966796875 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5038594007492065 6 PurityFd§à/0FRt0SRqqt2FBt/FBt_purity‚0.992598831653595 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 jet/accuracy_1_of_7‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9895125031471252 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.012544803321361542 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25269556045532227 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.046810582280159 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9850271344184875 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.4080808162689209 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.31172841787338257 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20752665400505066 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.37848320603370667 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963724613189697 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4165150225162506 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.001855287584476173 particle/accuracy_1_of_4‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01446305401623249 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3888888657093048 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3028053045272827 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.013926682062447071 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973641633987427 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.380405992269516 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2541394531726837 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.012022480368614197 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.990384578704834 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 particle/accuracy_7_of_7‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.37070873379707336 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9918761849403381 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9914723634719849 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2718396484851837 particle/accuracy_4_of_5‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.23603633046150208 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1226029098033905 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2568306028842926 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_1_of_7‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2413385808467865 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41712918877601624 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25967252254486084 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29670700430870056 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3583333492279053 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.3020086884498596 ("Purity/2FRt2SRqqt0FBt/event_purity‚0¨%%.ø‚(:„ §ðèÔ—Ýì¬ gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ĵõ˜zù‚(:ï §ðèÔºÕë®$ disk.nvme0n1.out‚97812.005859375" memory_percent‚96.584153175354( proc.memory.availableMB‚ 559.65234375 disk.sda1.out‚0( disk./.usagePercent‚78.7691479811205$ disk./.usageGB‚1173.717227935791# disk.nvme0c0n1.in‚ 18010.2578125! disk.nvme0n1.in‚ 18010.2578125 disk.dm-0.in‚ 18010.2578125# proc.memory.rssMB‚ 2936.12109375* proc.memory.percent‚17.920660972595215 disk.drbd1260.out‚ 0.57421875 disk.sda.in‚0 disk.sda2.in‚0& disk.nvme0c0n1.out‚97812.005859375 cpu‚25.226041099547686 proc.cpu.threads‚17 network.sent‚ 8.286692e+06 disk.sda3.out‚ 115.33984375 disk.dm-1.out‚ 115.33984375 disk.drbd1260.in‚0 disk.sda.out‚ 115.33984375 disk.sda1.in‚0 disk.sda2.out‚0! disk.dm-0.out‚97812.005859375 network.recv‚ 1.323218e+06 disk.sda3.in‚0 disk.dm-1.in‚0M¹dú‚(:’ ¶ðèÔŽëã® gpu.0.gpu‚47.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.585' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.717000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F)ÎQ|û‚(:ñ ¶ðèÔíü„° disk.dm-0.in‚ 18010.2578125 disk.sda3.in‚0 disk.sda3.out‚ 115.9609375 disk.sda1.in‚0 disk.sda2.out‚0 network.sent‚ 8.411911e+06' disk.nvme0c0n1.out‚98616.3564453125% disk.nvme0n1.out‚98616.3564453125 disk.sda1.out‚0# proc.memory.rssMB‚ 2935.12109375* proc.memory.percent‚17.914557456970215 proc.cpu.threads‚17! disk.nvme0n1.in‚ 18010.2578125" disk.dm-0.out‚98616.3564453125 disk.dm-1.out‚ 115.9609375 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 115.9609375 disk.sda2.in‚0$ memory_percent‚93.16418170928955# disk.nvme0c0n1.in‚ 18010.2578125 disk.dm-1.in‚0 disk.drbd1260.out‚0.578125) proc.memory.availableMB‚ 1119.98046875 cpu‚25.22443194775839 network.recv‚ 1.328665e+06) disk./.usagePercent‚78.83388537432643% disk./.usageGB‚1174.6818618774414Eê¬Â2ü- _wandbruntime‚3942  _runtime‚3942ª¯§àÇýÁ & lr-AdamW/pg1‚0.0006979439438665072 & lr-AdamW/pg2‚0.0006979439438665072 trainer/global_step‚3649 % _timestamp‚1.7884918431231718e+09 _runtime‚3942.763675255 _step‚297é}øÛÌþÆ trainer/global_step‚3649 % _timestamp‚1.7884918431231718e+09 _runtime‚3942.763675255 _step‚297 & lr-AdamW/pg1‚0.0006979439438665072 & lr-AdamW/pg2‚0.0006979439438665072©15ˆ2ÿ- _wandbruntime‚3942  _runtime‚3942„VDnò'€ìO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚255719370063360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255719352060416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249154929546576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚283918829275136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64050880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚3942.765083809 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3755996505088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3755996505088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36729835 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3922022211584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3742474711216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3743245359628 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57191768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36729837 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64050880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81055624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57191225 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64050878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259476139213312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145106502 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 % _timestamp‚1.7884918431245139e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚255719352060416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287840827546624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚283918805335040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚252898191945564 _step‚298 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249154946585936 trainer/global_step‚3649 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243248128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3756769149952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20461390 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20461931 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255719370063360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3922241519616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259475348565504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3756769149952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81055624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259475348565504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81078640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145129520 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219308032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0™ñ5Ï÷'ñO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81078640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚252897404257792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81055624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259475348565504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81078640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145129520 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 219308032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚255719370063360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255719352060416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249154929546576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚283918829275136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64050880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚3942.765083809 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3755996505088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3755996505088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36729835 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3922022211584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3742474711216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3743245359628 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57191768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36729837 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64050880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81055624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57191225 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64050878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259476139213312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145106502 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 % _timestamp‚1.7884918431245139e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚255719352060416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287840827546624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚283918805335040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚252898191945564 _step‚298 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249154946585936 trainer/global_step‚3649 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243248128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3756769149952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20461390 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20461931 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255719370063360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3922241519616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259475348565504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3756769149952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255787051999744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145145087 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36739483 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81077335 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 J=Devi R˜MkceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64067752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283994153677824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145168131 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259544843911680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64067752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249221013021696 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3743475851094 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚255787070002688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3757773908992 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57206066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚287917207133184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208808960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚252964471833430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81100377 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3757001250816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81077335 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3923262264320 trainer/global_step‚3649 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚283994177617920 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚255787070002688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145168131 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3757773908992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36739485 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64067754Z«p'„ûM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259544053250560 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚287917439882240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚3943.750526959 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57206066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚287917207133184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208808960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚252964471833430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81100377 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3757001250816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81077335 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3923262264320 trainer/global_step‚3649 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚283994177617920 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚255787070002688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145168131 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3757773908992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36739485 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64067754 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232749056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255787051999744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145145087 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36739483 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81077335 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64067752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283994153677824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145168131 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259544843911680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64067752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249221013021696 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3743475851094 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚255787070002688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3757773908992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262910597232128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3792545296324 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82144526 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _step‚302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262911387879936 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _runtime‚3992.833613947 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82167542 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291660153443840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259104367029760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259104349026816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259104349026816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147062122 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64894580 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64894578 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3806248205312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚256251584318672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37212237 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82167542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚259104367029760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82144526 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚262910597232128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64894578 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825ÌÇuº÷'•ñO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64894578 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3806248205312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚256251584318672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37212237 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82167542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚259104367029760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82144526 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚262910597232128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64894578 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚252458268373936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚252458251334576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3793315944736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147039104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262910597232128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3792545296324 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82144526 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _step‚302 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262911387879936 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _runtime‚3992.833613947 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82167542 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291660153443840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259104367029760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259104349026816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259104349026816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147062122 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64894580 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489®¢ Fq2–- _wandbruntime‚3993  _runtime‚3993ó‘û&—õM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3808025609216 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82189279 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57948879 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚256317864206538 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64911454 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82166237 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20726473 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚259172048966144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3975169131008 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3807252951040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3808025609216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287761594757120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287761570817024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚252524317770336 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37221883 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291736763888128 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64911452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82166237 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57948356 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20726994 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚3699 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3807252951040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147100733 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259172066969088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚259172048966144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252524334809696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 $ _timestamp‚1.788491894270975e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3794317084822 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 _runtime‚3993.912285077 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64911452 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚262980092578304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147077689 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3974964516352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147100733 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147077689 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚262980092578304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37221883 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291736763888128 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64911452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82166237 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57948356 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20726994 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚3699 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3807252951040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147100733 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259172066969088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚259172048966144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252524334809696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 $ _timestamp‚1.788491894270975e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3794317084822 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 _runtime‚3993.912285077 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64911452 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚262980092578304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147077689 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3974964516352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147100733 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147077689 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚262980092578304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82189279 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57948879 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚256317864206538 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64911454 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82166237 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20726473 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚259172048966144 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3975169131008 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¯G¡ËÊ2™- _wandbruntime‚3993  _runtime‚3993bŵ”ˆšÒ 2loss/SRqqt3/assignment_loss‚1.4544059212848142 /loss/FRt2/detection_loss‚2.4971926302620857 /loss/FBt1/assignment_loss¶–ÍЂ0.031966084940359 1loss/FBt3/assignment_loss‚0.15391588743077592 _step‚304 $ _timestamp‚1.788491894274063e+09 0loss/FRt2/assignment_loss‚3.5004038232745547 /loss/FRt3/detection_loss‚2.4821008425709365 epoch‚3 /loss/FRt1/assignment_loss‚2.694817768900018 0loss/FRt4/assignment_loss‚1.3419285895777684 1loss/SRqqt1/assignment_loss‚1.906849182484781 2loss/FBt2/assignment_loss‚0.004501301470944392 1loss/FBt4/assignment_loss‚0.09981402847481408 .loss/FRt1/detection_loss‚2.665078508226495 /loss/FRt4/detection_loss‚1.5768863208518893 2loss/SRqqt4/assignment_loss‚0.7708890941840681 1loss/SRqqt1/detection_loss‚2.3509857374079086 1loss/SRqqt3/detection_loss‚1.4801141962006286 .loss/FBt2/detection_loss‚2.678730172770364 .loss/FBt3/detection_loss‚2.420869600772858 %loss/total_loss‚42.17903821700823 /loss/FRt3/assignment_loss‚2.563488220846331 2loss/SRqqt2/assignment_loss‚2.1829032471234147 0loss/SRqqt2/detection_loss‚2.774329450997439 .loss/FBt1/detection_loss‚3.067304563522339 _runtime‚3993.914320152 .loss/FBt4/detection_loss‚0.613833285424681 trainer/global_step‚3699 1loss/SRqqt4/detection_loss‚0.8657297580079599Ò =]›× $ _timestamp‚1.788491894274063e+09 0loss/FRt2/assignment_loss‚3.5004038232745547 /loss/FRt3/detection_loss‚2.4821008425709365 epoch‚3 /loss/FRt1/assignment_loss‚2.694817768900018 0loss/FRt4/assignment_loss‚1.3419285895777684 1loss/SRqqt1/assignment_loss‚1.906849182484781 2loss/FBt2/assignment_loss‚0.004501301470944392 1loss/FBt4/assignment_loss‚0.09981402847481408 .loss/FRt1/detection_loss‚2.665078508226495 /loss/FRt4/detection_loss‚1.5768863208518893 2loss/SRqqt4/assignment_loss‚0.7708890941840681 1loss/SRqqt1/detection_loss‚2.3509857374079086 1loss/SRqqt3/detection_loss‚1.4801141962006286 .loss/FBt2/detection_loss‚2.678730172770364 .loss/FBt3/detection_loss‚2.420869600772858 %loss/total_loss‚42.17903821700823 /loss/FRt3/assignment_loss‚2.563488220846331 2loss/SRqqt2/assignment_loss‚2.1829032471234147 0loss/SRqqt2/detection_loss‚2.774329450997439 .loss/FBt1/detection_loss‚3.067304563522339 _runtime‚3993.914320152 .loss/FBt4/detection_loss‚0.613833285424681 trainer/global_step‚3699 1loss/SRqqt4/detection_loss‚0.8657297580079599 2loss/SRqqt3/assignment_loss‚1.4544059212848142 /loss/FRt2/detection_loss‚2.4971926302620857 /loss/FBt1/assignment_loss‚0.031966084940359 1loss/FBt3/assignment_loss‚0.15391588743077592 _step‚304°}‹˜’œ‚(:‡ ñèÔþ² gpu.0.gpu‚35.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.083' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sÈ9)‚(:ô ñèÔŒ“ƒ¸ disk.sda3.out‚ 118.05859375 disk.dm-1.out‚ 118.05859375 disk.sda.in‚0 network.sent‚ 8.540259e+06 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.5859375 disk.sda.out‚ 118.05859375 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚93.16277503967285( proc.memory.availableMB‚ 1120.2109375 disk.drbd1260.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚100971.0283203125 disk.nvme0n1.in‚ 18172.265625& disk.nvme0n1.out‚100971.0283203125* proc.memory.percent‚17.914557456970215 proc.cpu.threads‚17# disk.dm-0.out‚100971.0283203125 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 18172.265625# proc.memory.rssMB‚ 2935.12109375 cpu‚25.222292422075377 network.recv‚ 1.351133e+06( disk./.usagePercent‚78.8437007031892% disk./.usageGB‚1174.8281173706055 disk.dm-0.in‚ 18172.265625 disk.sda3.in‚0J’ò’ž‚(:‡ ñèÔÔ˯ gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.994' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ïq,„Ÿ‚(:ù ñèÔ­œŒ³ network.recv‚ 1.357711e+06 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚101338.88330078125$ disk.dm-0.out‚101338.88330078125( proc.memory.availableMB‚ 1120.6953125 disk.dm-1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 18172.265625$ memory_percent‚93.15981864929199# proc.memory.rssMB‚ 2935.12109375* proc.memory.percent‚17.914557456970215 cpu‚25.221844928364487 disk.sda3.out‚ 118.55078125 disk.drbd1260.out‚ 0.58984375 disk.sda.in‚0 disk.sda.out‚ 118.55078125 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 18172.265625' disk.nvme0n1.out‚101338.88330078125 disk.dm-0.in‚ 18172.265625 network.sent‚ 8.648751e+06) disk./.usagePercent‚78.84212625685363% disk./.usageGB‚1174.8046569824219 disk.dm-1.out‚ 118.55078125 proc.cpu.threads‚17 disk.sda1.out‚0É%9³ ‚(:… ŸñèԬ͖­ gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶“¥]|¡‚(:ñ ŸñèÔ³¢œ° cpu‚25.22149787484357 disk.dm-1.out‚ 118.89453125 disk.drbd1260.out‚ 0.58984375 disk.sda2.out‚0$ memory_percent‚93.15824508666992# proc.memory.rssMB‚ 2934.12109375 proc.cpu.threads‚17 network.sent‚ 8.653898e+06 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 118.89453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 18387.1875) disk./.usagePercent‚78.84370479930976% disk./.usageGB‚1174.8281784057617 disk.nvme0c0n1.in‚ 18387.1875' disk.nvme0n1.out‚101807.82470703125$ disk.dm-0.out‚101807.82470703125 disk.sda3.in‚0 disk.sda3.out‚ 118.89453125* proc.memory.percent‚17.908453941345215 network.recv‚ 1.361478e+06 disk.sda2.in‚0) disk.nvme0c0n1.out‚101807.82470703125 disk.nvme0n1.in‚ 18387.1875' proc.memory.availableMB‚ 1120.953125­ð¯2¢- _wandbruntime‚4043  _runtime‚4043âA~xÅ£¿ _step‚305 % lr-AdamW/pg1‚0.000690412170210851 % lr-AdamW/pg2‚0.000690412170210851 trainer/global_step‚3749 % _timestamp‚1.7884919438405995e+09 _runtime‚4043.481016428àa\Ê¤Ä % lr-AdamW/pg1‚0.000690412170210851 % lr-AdamW/pg2‚0.000690412170210851 trainer/global_step‚3749 % _timestamp‚1.7884919438405995e+09 _runtime‚4043.481016428 _step‚305±V1Å2¥- _wandbruntime‚4043  _runtime‚4043ðà·<ò'¦ìO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚266345845898752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20983552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83256444 _step‚306 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚262489345993216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83233428 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295479957064704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266345845898752 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚266346636546560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58678710 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65738278 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83256444 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183656448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259604189004008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3843386529844 % _timestamp‚1.7884919438419693e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148971706 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚255761590161936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3842615881432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚262489345993216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚266346636546560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207596544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4026109854720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3856499905536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65738280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚262489363996160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65738278 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚259604976691780 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3856499905536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291453663553536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3749 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148994724 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4026293511168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37694635 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291453639613440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37694637 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005lZž]÷'§ñO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259604189004008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3843386529844 % _timestamp‚1.7884919438419693e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148971706 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚255761590161936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3842615881432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚262489345993216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚266346636546560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207596544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4026109854720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3856499905536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65738280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚262489363996160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65738278 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚259604976691780 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3856499905536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291453663553536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3749 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148994724 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4026293511168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37694635 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291453639613440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37694637 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚266345845898752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20983552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83256444 _step‚306 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚262489345993216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83233428 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295479957064704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266345845898752 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚266346636546560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58678710 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65738278 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83256444 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183656448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040²äö©2¨- _wandbruntime‚4044  _runtime‚4044÷ l "©öM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4027351262208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65755152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149033335 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83255139 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259671256579646 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255827656597696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20989308 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83278181 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3857504651264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37704285 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65755154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3858277309440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚262557045932544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4027142453248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚266415341244928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149033335 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚262557063935488 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149010291 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58693066 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3858277309440 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149010291 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83255139 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232749056 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295556130409472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295556363158528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259672044267626 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20988783 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3843617021310 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚266414550583808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚262557063935488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58693593 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291528987956224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 _step‚307 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚4044.514399287 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83278181 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005ÉÑO'ªûM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20988783 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3843617021310 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚266414550583808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚262557063935488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58693593 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291528987956224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 _step‚307 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚4044.514399287 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83278181 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4027351262208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65755152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149033335 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83255139 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259671256579646 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255827656597696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20989308 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83278181 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3857504651264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37704285 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65755154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3858277309440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚262557045932544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4027142453248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚266415341244928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149033335 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚262557063935488 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149010291 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58693066 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3858277309440 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149010291 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83255139 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232749056 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295556130409472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295556363158528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259672044267626³0U2«- _wandbruntime‚4044  _runtime‚4044—jD<[¬Õ 2loss/SRqqt4/assignment_loss‚0.8977840344111124 -loss/FRt2/detection_loss‚2.42841222612215 1loss/FBt4/assignment_loss‚0.06580180755673821 2loss/SRqqt3/assignment_loss‚1.4967520090181436 1loss/FBt2/assignment_loss‚0.44957866542972624 /loss/FBt2/detection_loss‚3.7418070236841836 %loss/total_loss‚43.44551450113984 0loss/FRt1/assignment_loss‚3.2141033151210885 0loss/SRqqt1/detection_loss‚2.192182950484447 /loss/FRt2/assignment_loss‚3.559327718886462 0loss/FRt4/assignment_loss‚1.2654390454292297 1loss/SRqqt2/assignment_loss‚1.620553344488144 1loss/FBt3/assignment_loss‚0.10978074639489124 .loss/FBt3/detection_loss‚3.110524641142951 1loss/FBt1/assignment_loss‚0.22668873110438248 1loss/SRqqt3/detection_loss‚1.4617135971228579 1loss/SRqqt4/detection_loss‚1.0987842783095345 .loss/FBt1/detection_loss‚2.710838649008009 /loss/FBt4/detection_loss‚0.6168134512119395 trainer/global_step‚3749 _step‚308 0loss/FRt3/assignment_loss‚2.4870151587021656 2loss/SRqqt1/assignment_loss‚1.5363445345025797 /loss/FRt4/detection_loss‚1.4934853875388703 0loss/SRqqt2/detection_loss‚2.522468055997576 epoch‚3 % _timestamp‚1.7884919448746793e+09 _runtime‚4044.515803193 /loss/FRt1/detection_loss‚2.7734437397657294 /loss/FRt3/detection_loss‚2.3658713897069297¤Ó}Œ`­Ú 2loss/SRqqt4/assignment_loss‚0.8977840344111124 -loss/FRt2/detection_loss‚2.42841222612215 1loss/FBt4/assignment_loss‚0.06580180755673821 2loss/SRqqt3/assignment_loss‚1.4967520090181436 1loss/FBt2/assignment_loss‚0.44957866542972624 /loss/FBt2/detection_loss‚3.7418070236841836 %loss/total_loss‚43.44551450113984 0loss/FRt1/assignment_loss‚3.2141033151210885 0loss/SRqqt1/detection_loss‚2.192182950484447 /loss/FRt2/assignment_loss‚3.559327718886462 0loss/FRt4/assignment_loss‚1.2654390454292297 1loss/SRqqt2/assignment_loss‚1.620553344488144 1loss/FBt3/assignment_loss‚0.10978074639489124 .loss/FBt3/detection_loss‚3.110524641142951 1loss/FBt1/assignment_loss‚0.22668873110438248 1loss/SRqqt3/detection_loss‚1.4617135971228579 1loss/SRqqt4/detection_loss‚1.0987842783095345 .loss/FBt1/detection_loss‚2.710838649008009 /loss/FBt4/detection_loss‚0.6168134512119395 trainer/global_step‚3749 _step‚308 0loss/FRt3/assignment_loss‚2.4870151587021656 2loss/SRqqt1/assignment_loss‚1.5363445345025797 /loss/FRt4/detection_loss‚1.4934853875388703 0loss/SRqqt2/detection_loss‚2.522468055997576 epoch‚3 % _timestamp‚1.7884919448746793e+09 _runtime‚4044.515803193 /loss/FRt1/detection_loss‚2.7734437397657294 /loss/FRt3/detection_loss‚2.3658713897069297´l¾a’®‚(:‡ ®ñèÔô°ï¯ gpu.0.gpu‚36.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’1…v¯‚(:ë ®ñèÔªæ³ disk.sda2.out‚0$ memory_percent‚93.15540790557861* proc.memory.percent‚17.908453941345215 network.sent‚ 8.658941e+06) proc.memory.availableMB‚ 1121.41796875# proc.memory.rssMB‚ 2934.12109375 proc.cpu.threads‚17 network.recv‚ 1.364854e+06 disk.sda1.out‚0# disk.dm-0.out‚102246.8447265625 disk.sda3.in‚0 disk.dm-1.out‚ 119.3671875 disk.drbd1260.in‚0 disk.drbd1260.out‚0.59375 cpu‚25.221142155888113 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 18387.1875( disk.nvme0c0n1.out‚102246.8447265625 disk.nvme0n1.in‚ 18387.1875& disk.nvme0n1.out‚102246.8447265625 disk.dm-0.in‚ 18387.1875 disk.sda3.out‚ 119.3671875 disk.dm-1.in‚0) disk./.usagePercent‚78.84027916249408% disk./.usageGB‚1174.7771339416504 disk.sda.in‚0 disk.sda.out‚ 119.3671875 disk.sda1.in‚0UœÕq‘°‚(:† ½ñèÔ¿Ýϯ gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.945 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<’./x±‚(:í ½ñèÔ‚”¿´ network.sent‚ 8.767976e+06% disk./.usageGB‚1174.7771339416504 disk.sda.out‚ 119.76953125 disk.sda1.in‚0# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 proc.cpu.threads‚17 network.recv‚ 1.371697e+06& disk.nvme0n1.out‚102618.5830078125 disk.dm-0.in‚ 18387.1875# disk.dm-0.out‚102618.5830078125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0) disk./.usagePercent‚78.84027916249408 disk.nvme0n1.in‚ 18387.1875 disk.sda3.out‚ 119.76953125 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 18387.1875$ memory_percent‚93.15543174743652( proc.memory.availableMB‚ 1121.4140625 disk.dm-1.out‚ 119.76953125 disk.drbd1260.out‚0.59375 disk.sda2.in‚0( disk.nvme0c0n1.out‚102618.5830078125 cpu‚25.220745565432164irÎÝ’²‚(:‡ ÌñèÔåñ—± gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.012' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z䳂(:ô ÌñèÔ­•å¸ disk.dm-0.in‚ 18387.1875 disk.dm-1.out‚ 120.19140625 disk.sda2.in‚0' disk.nvme0n1.out‚103009.58349609375$ memory_percent‚93.15845966339111) proc.memory.availableMB‚ 1120.91796875* proc.memory.percent‚17.908453941345215 cpu‚25.220315905276024) disk./.usagePercent‚78.84028223458449 disk.dm-1.in‚0) disk.nvme0c0n1.out‚103009.58349609375# proc.memory.rssMB‚ 2934.12109375 proc.cpu.threads‚17 network.recv‚ 1.375975e+06 disk.sda3.in‚0 disk.sda3.out‚ 120.19140625 disk.sda.in‚0 disk.sda.out‚ 120.19140625 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 18387.1875$ disk.dm-0.out‚103009.58349609375 network.sent‚ 8.773828e+06% disk./.usageGB‚1174.7771797180176 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.59765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 18387.1875}k2´- _wandbruntime‚4094  _runtime‚4094|Ò~GǵÁ _step‚309 & lr-AdamW/pg1‚0.0006865538930027042 & lr-AdamW/pg2‚0.0006865538930027042 trainer/global_step‚3799 % _timestamp‚1.7884919943940828e+09 _runtime‚4094.034460301ÖwåÌ¶Æ & lr-AdamW/pg1‚0.0006865538930027042 & lr-AdamW/pg2‚0.0006865538930027042 trainer/global_step‚3799 % _timestamp‚1.7884919943940828e+09 _runtime‚4094.034460301 _step‚309µbvä!2·- _wandbruntime‚4094  _runtime‚4094„U_\ò'¸ìO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59421644 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295221080692736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _step‚310 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265874342959616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 % _timestamp‚1.7884919943954134e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚262957581377116 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84345346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259064911949936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269781094565376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84322330 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21244607 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269781094565376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269781885213184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262958369064888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66581980 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚3799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4078157713920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66581978 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3906751605760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59421111 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38177037 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299299214466560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66581980 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84322330 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21244076 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150927326 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3892686466540 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0to÷Þ¶¹ñO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84345346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚299299430451712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.Z»þ›A&on_train_batch_start/requested_bytes.all.current‚ 787687772 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66581980 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚3799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4078157713920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66581978 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3906751605760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59421111 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38177037 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299299214466560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66581980 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84322330 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21244076 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150927326 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3892686466540 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59421644 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295221080692736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _step‚310 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265874342959616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 % _timestamp‚1.7884919943954134e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚262957581377116 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84345346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259064911949936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269781094565376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84322330 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21244607 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269781094565376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269781885213184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262958369064888 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150965937 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4079397505536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232749056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295296405095424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚311 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66598854 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263025436640734 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269850589911552 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38186685 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59436022 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259130961346336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84344041 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 trainer/global_step‚3799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66598852 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38186683 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150942893 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3894458255038 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3907756351488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265942060901888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21249339 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21249870 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150942893 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66598854 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295296429035520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59436555 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299375593792000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208808960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚269850589911552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 $ _timestamp‚1.788491995320108e+09 _runtime‚4094.961122501 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150965937 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84344041 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84367083 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265942060901888 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263024648952754 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59436555 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299375593792000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208808960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚269850589911552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 $ _timestamp‚1.788491995320108e+09 _runtime‚4094.961122501 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150965937 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84344041 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84367083 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265942060901888 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263024648952754 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150965937 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4079397505536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232749056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295296405095424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚311 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66598854 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263025436640734 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269850589911552 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38186685 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59436022 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259130961346336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84344041 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 trainer/global_step‚3799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66598852 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38186683 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150942893 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3894458255038 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3907756351488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265942060901888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21249339 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21249870 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150942893 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66598854 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295296429035520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489·CÂE2½- _wandbruntime‚4094  _runtime‚4094 2s]¾× /loss/FRt2/assignment_loss‚3.092447890954859 .loss/FRt1/detection_loss‚2.498440387845039 1loss/SRqqt3/detection_loss‚1.3847389777361507 _runtime‚ 4094.96312223 .loss/FBt4/detection_loss‚0.465739851374823 epoch‚3 1loss/FBt2/assignment_loss‚0.26053645408501325 /loss/FRt4/detection_loss‚1.6263544278448088 2loss/SRqqt4/assignment_loss‚0.8669606371187284 .loss/FRt3/detection_loss‚2.218488335609436 1loss/SRqqt2/detection_loss‚2.0105473682993935 /loss/FBt1/detection_loss‚3.9659469723701477 .loss/FBt3/detection_loss‚2.262782722711563 &loss/total_loss‚40.418856787235896 /loss/FRt2/detection_loss‚2.4016440878900784 /loss/FRt1/assignment_loss‚3.403902043615069 0loss/FRt4/assignment_loss‚1.5822703995202716 2loss/SRqqt3/assignment_loss‚1.6224548914010677 2loss/FBt3/assignment_loss‚0.042753433963904776 1loss/FBt4/assignment_loss‚0.13599086268350796 /loss/FBt2/detection_loss‚1.8297918505138822 /loss/FRt3/assignment_loss‚2.355098082260652 2loss/SRqqt2/assignment_loss‚1.4754691718234902 1loss/SRqqt1/detection_loss‚2.2530126293500263 1loss/SRqqt4/detection_loss‚0.8635796102099731 trainer/global_step‚3799 % _timestamp‚1.7884919953229332e+09 _step‚312 2loss/SRqqt1/assignment_loss‚1.6406307766834896 1loss/FBt1/assignment_loss‚0.15927492137052468Û¢Áb¿Ü .loss/FBt4/detection_loss‚0.465739851374823 epoch‚3 1loss/FBt2/assignment_loss‚0.26053645408501325 /loss/FRt4/detection_loss‚1.6263544278448088 2loss/SRqqt4/assignment_loss‚0.8669606371187284 .loss/FRt3/detection_loss‚2.218488335609436 1loss/SRqqt2/detection_loss‚2.0105473682993935 /loss/FBt1/detection_loss‚3.9659469723701477 .loss/FBt3/detection_loss‚2.262782722711563 &loss/total_loss‚40.418856787235896 /loss/FRt2/detection_loss‚2.4016440878900784 /loss/FRt1/assignment_loss‚3.403902043615069 0loss/FRt4/assignment_loss‚1.5822703995202716 2loss/SRqqt3/assignment_loss‚1.6224548914010677 2loss/FBt3/assignment_loss‚0.042753433963904776 1loss/FBt4/assignment_loss‚0.13599086268350796 /loss/FBt2/detection_loss‚1.8297918505138822 /loss/FRt3/assignment_loss‚2.355098082260652 2loss/SRqqt2/assignment_loss‚1.4754691718234902 1loss/SRqqt1/detection_loss‚2.2530126293500263 1loss/SRqqt4/detection_loss‚0.8635796102099731 trainer/global_step‚3799 % _timestamp‚1.7884919953229332e+09 _step‚312 2loss/SRqqt1/assignment_loss‚1.6406307766834896 1loss/FBt1/assignment_loss‚0.15927492137052468 /loss/FRt2/assignment_loss‚3.092447890954859 .loss/FRt1/detection_loss‚2.498440387845039 1loss/SRqqt3/detection_loss‚1.3847389777361507 _runtime‚ 4094.96312223¸ÙñKiÀ‚(:’ ÛñèÔ°ûæ¬ gpu.0.gpu‚49.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.794998168‡6´494531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.004' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.600799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«Jmâ„Á‚(:ù ÛñèÔ»ÎÜ® disk.sda2.in‚0$ disk.dm-0.out‚103328.36474609375 disk.sda3.in‚0 cpu‚25.21989020935349 network.recv‚ 1.379671e+06) disk./.usagePercent‚78.83871239638454 disk.sda.out‚ 120.51953125 disk.sda2.out‚0 disk.dm-0.in‚ 18508.859375 disk.sda3.out‚ 120.51953125 disk.dm-1.in‚0$ memory_percent‚93.15493106842041 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 18508.859375 disk.nvme0n1.in‚ 18508.859375' disk.nvme0n1.out‚103328.36474609375# proc.memory.rssMB‚ 2934.12109375 network.sent‚ 8.778921e+06 disk.sda.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚103328.36474609375 disk.dm-1.out‚ 120.51953125) proc.memory.availableMB‚ 1121.49609375* proc.memory.percent‚17.908453941345215 proc.cpu.threads‚17% disk./.usageGB‚1174.7537879943848 disk.drbd1260.out‚ 0.59765625 disk.sda1.in‚0ÊdQ‚(:’ êñèÔ¬Öö¬ gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.028' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.005599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vyãÒvÂ(:ë êñèÔÒÚ½® network.sent‚ 8.886646e+06 disk.nvme0n1.in‚ 18508.859375$ disk.nvme0n1.out‚103912.58984375 disk.drbd1260.out‚ 0.6015625 disk.sda.out‚ 120.9609375# memory_percent‚89.8960828781128 proc.cpu.threads‚17 disk.sda3.out‚ 120.9609375% disk./.usageGB‚1174.1410102844238 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1655.42578125# proc.memory.rssMB‚ 2934.12109375 network.recv‚ 1.385453e+06) disk./.usagePercent‚78.79758837011431 disk.dm-1.out‚ 120.9609375 disk.sda1.in‚0* proc.memory.percent‚17.908453941345215" disk.nvme0c0n1.in‚ 18508.859375& disk.nvme0c0n1.out‚103912.58984375 disk.dm-0.in‚ 18508.859375! disk.dm-0.out‚103912.58984375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 cpu‚25.21934479656567yÝZœÄ‚(:‘ ùñèÔ¹£¥­ gpu.0.gpu‚34.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.128' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.22559999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_õ|zÅ‚(:ï ùñèÔ—œû®" disk.nvme0c0n1.in‚ 18508.859375# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215% disk./.usageGB‚1174.1410102844238 disk.sda2.out‚0 disk.nvme0n1.in‚ 18508.859375( proc.memory.availableMB‚ 1559.1953125 cpu‚25.218794272042256) disk./.usagePercent‚78.79758837011431 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0& disk.nvme0c0n1.out‚104188.18359375 proc.cpu.threads‚17 network.sent‚ 8.893097e+06 disk.dm-0.in‚ 18508.859375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.6015625 disk.sda.out‚ 121.39453125$ disk.nvme0n1.out‚104188.18359375$ memory_percent‚90.48342704772949 network.recv‚ 1.390361e+06! disk.dm-0.out‚104188.18359375 disk.sda3.out‚ 121.39453125 disk.dm-1.in‚0 disk.dm-1.out‚ 121.39453125Äk°)Æ‚(:’ ˆòèÔþ¸µ­ gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚159.242' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.848399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ÑnWxÇ‚(:í ˆòèÔîçÙ¯ disk.dm-1.out‚ 121.8046875 disk.sda.in‚0* proc.memory.percent‚17.908453941345215 cpu‚25.21824422912734 proc.cpu.threads‚17% disk./.usageGB‚1174.1410331726074 disk.sda3.in‚0 disk.sda3.out‚ 121.8046875 disk.drbd1260.out‚0.609375 disk.sda1.in‚0 disk.nvme0n1.in‚ 18508.859375% disk.nvme0n1.out‚104549.259765625 network.recv‚ 1.394867e+06) disk./.usagePercent‚78.79758990615952 disk.sda.out‚ 121.8046875 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 18508.859375' disk.nvme0c0n1.out‚104549.259765625$ memory_percent‚91.02351665496826 network.sent‚ 8.90088e+06 disk.dm-0.in‚ 18508.859375" disk.dm-0.out‚104549.259765625 disk.drbd1260.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1470.70703125# proc.memory.rssMB‚ 2934.12109375 disk.dm-1.in‚0Ÿ=!ì2È- _wandbruntime‚4144  _runtime‚4144ü¯UÇÉÁ _runtime‚4144.059034394 _step‚313 & lr-AdamW/pg1‚0.0006826350900262467 & lr-AdamW/pg2‚0.0006826350900262467 trainer/global_step‚3849 % _timestamp‚1.7884920444187217e+09…<”ÌÊÆ trainer/global_step‚3849 % _timestamp‚1.7884920444187217e+09 _runtime‚4144.059034394 _step‚313 & lr-AdamW/pg1‚0.0006826350900262467 & lr-AdamW/pg2‚0.0006826350900262467¹ëJ¸¦2Ë- _wandbruntime‚4144  _runtime‚4144qŽaò'ÌìO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚269259339926016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67425678 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚273217133879808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3849 % _timestamp‚1.7884920444197187e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273216343232000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚266310973750224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚269259357928960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60166260 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303119001673216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21507366 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4130503841280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67425680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85411232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4130305504768 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _step‚314 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3957003305984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67425680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60166803 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67425678 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303118779396608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85434248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298988497831936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚266311761437996 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273216343232000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269259339926016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85434248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269259357928960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 152859928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3942757051648 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273216343232000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚266310973750224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚269259357928960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60166260 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303119001673216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21507366 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4130503841280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67425680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85411232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4130305504768 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _step‚314 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3957003305984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67425680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60166803 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67425678 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303118779396608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85434248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298988497831936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚266311761437996ºwÓy2Î- _wandbruntime‚4144  _runtime‚4144P€‹ˆÏöM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38669085 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273285838578176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3958008051712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152875495 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚315 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3958008051712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60181657 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67442552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21512572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3944528840146 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262434283134336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152898539 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152898539 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚303195153762816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299063846174720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3958780709888 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60181126 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4131331528192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚269327057868288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262434300173696 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3958780709888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273285047917056 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38669083 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚266378829013842 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 @p*¼Ùt :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269327039865344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210906112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273285838578176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67442554 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85455985 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67442554 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3943758191526 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3849 % _timestamp‚1.7884920452855322e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67442552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266378041325862 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚4144.926599255 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4131542434304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299063822234624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85432943 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269327039865344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269327039865344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210906112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273285838578176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67442554 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85455985 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67442554 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3943758191526 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3849 % _timestamp‚1.7884920452855322e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67442552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266378041325862 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚4144.926599255 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4131542434304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299063822234624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85432943 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269327039865344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚315 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3958008051712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60181657 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67442552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21512572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3944528840146 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262434283134336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152898539 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152898539 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚303195153762816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299063846174720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3958780709888 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60181126 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4131331528192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620»ï£!Z2Ñ- _wandbruntime‚4144  _runtime‚4144Ø¿8 ZÒÔ 0loss/FBt1/assignment_loss‚0.6652320883236825 1loss/FBt4/assignment_loss‚0.06117926625643703 epoch‚3 /loss/FRt3/assignment_loss‚2.710075753239485 2loss/SRqqt3/assignment_loss‚1.4140633910995077 0loss/FBt2/assignment_loss‚0.3658476064374554 0loss/SRqqt1/detection_loss‚2.319376817520927 0loss/FRt2/assignment_loss‚2.6239483529521572 0loss/FRt4/assignment_loss‚1.3447539765726437 2loss/SRqqt4/assignment_loss‚1.2328375701326877 /loss/FBt2/detection_loss‚2.2473558858036995 trainer/global_step‚3849 0loss/FRt1/assignment_loss‚3.5356183065924536 1loss/FBt3/assignment_loss‚0.41652984796091913 /loss/FRt1/detection_loss‚2.3590275215548138 1loss/SRqqt3/detection_loss‚1.4521243880326684 /loss/FRt2/detection_loss‚2.2688996618834345 1loss/SRqqt1/assignment_loss‚1.994877061861403 0loss/SRqqt2/detection_loss‚2.295328168996743 $loss/total_loss‚41.3955776249625 $ _timestamp‚1.788492045287861e+09 _step‚316 /loss/FRt4/detection_loss‚1.7912553618238731 /loss/FBt1/detection_loss‚2.0937853157520294 /loss/FBt4/detection_loss‚0.5855524042768904 _runtime‚4144.928033628 1loss/SRqqt2/assignment_loss‚1.795128156031881 /loss/FRt3/detection_loss‚2.1589324972950497 1loss/SRqqt4/detection_loss‚1.1718938979320228 /loss/FBt3/detection_loss‚2.4919543266296387ê—o_ÓÙ 1loss/SRqqt2/assignment_loss‚1.795128156031881 /loss/FRt3/detection_loss‚2.1589324972950497 1loss/SRqqt4/detection_loss‚1.1718938979320228 /loss/FBt3/detection_loss‚2.4919543266296387 0loss/FBt1/assignment_loss‚0.6652320883236825 1loss/FBt4/assignment_loss‚0.06117926625643703 epoch‚3 /loss/FRt3/assignment_loss‚2.710075753239485 2loss/SRqqt3/assignment_loss‚1.4140633910995077 0loss/FBt2/assignment_loss‚0.3658476064374554 0loss/SRqqt1/detection_loss‚2.319376817520927 0loss/FRt2/assignment_loss‚2.6239483529521572 0loss/FRt4/assignment_loss‚1.3447539765726437 2loss/SRqqt4/assignment_loss‚1.2328375701326877 /loss/FBt2/detection_loss‚2.2473558858036995 trainer/global_step‚3849 0loss/FRt1/assignment_loss‚3.5356183065924536 1loss/FBt3/assignment_loss‚0.41652984796091913 /loss/FRt1/detection_loss‚2.3590275215548138 1loss/SRqqt3/detection_loss‚1.4521243880326684 /loss/FRt2/detection_loss‚2.2688996618834345 1loss/SRqqt1/assignment_loss‚1.994877061861403 0loss/SRqqt2/detection_loss‚2.295328168996743 $loss/total_loss‚41.3955776249625 $ _timestamp‚1.788492045287861e+09 _step‚316 /loss/FRt4/detection_loss‚1.7912553618238731 /loss/FBt1/detection_loss‚2.0937853157520294 /loss/FBt4/detection_loss‚0.5855524042768904 _runtime‚4144.928033628¼ëkº£‘Ô‚(:† —òèÔ—©æ° gpu.0.gpu‚48.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.023 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀNSxÕ‚(:í —òèÔšï¾³! disk.nvme0c0n1.in‚ 18659.46875' disk.nvme0c0n1.out‚104910.044921875 disk.nvme0n1.in‚ 18659.46875( proc.memory.availableMB‚ 1366.9296875# proc.memory.rssMB‚ 2934.12109375 cpu‚25.217896638161474 disk.drbd1260.out‚0.609375 disk.sda.out‚ 122.08203125 disk.sda1.out‚0 disk.sda2.in‚0% disk.nvme0n1.out‚104910.044921875 disk.dm-0.in‚ 18659.46875 proc.cpu.threads‚17 network.recv‚ 1.398571e+06 disk.sda3.in‚0 disk.sda3.out‚ 122.08203125 disk.dm-1.out‚ 122.08203125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0" disk.dm-0.out‚104910.044921875) disk./.usagePercent‚78.79758760209171$ disk./.usageGB‚1174.140998840332 disk.drbd1260.in‚0$ memory_percent‚91.65692329406738* proc.memory.percent‚17.908453941345215 network.sent‚ 8.904108e+06 disk.dm-1.in‚0Óöª’Ö‚(:‡ ¦òèÔ­»Œ° gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.261' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6522 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…÷C\~ׂ(:ó ¦òèÔýÌì² disk.sda3.out‚ 122.53515625 disk.sda1.in‚0( disk.nvme0c0n1.out‚105103.7431640625 cpu‚25.217429193776507 network.sent‚ 9.011686e+06 network.recv‚ 1.403889e+06 disk.sda.out‚ 122.53515625& disk.nvme0n1.out‚105103.7431640625$ memory_percent‚92.32048988342285% disk./.usageGB‚1174.0474967956543 disk.dm-0.in‚ 18659.46875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.61328125 disk.sda1.out‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 18659.46875 disk.dm-1.out‚ 122.53515625 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 18659.46875( proc.memory.availableMB‚ 1258.2109375# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 proc.cpu.threads‚17) disk./.usagePercent‚78.79131260141995# disk.dm-0.out‚105103.7431640625 disk.sda3.in‚0ëa?ø’Ø‚(:‡ µòèÔ°ƒÌ­ gpu.0.gpu‚36.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.819' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5638 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý0‡3wÙ‚(:ì µòèÔÏ´ì° disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.908453941345215 proc.cpu.threads‚17 network.recv‚ 1.408911e+06 disk.sda3.out‚ 122.9609375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.61328125 disk.sda.out‚ 122.9609375( disk.nvme0c0n1.out‚105497.6064453125 disk.dm-0.in‚ 18659.46875) disk./.usagePercent‚78.80038909253891 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚105497.6064453125$ memory_percent‚92.90428161621094# proc.memory.rssMB‚ 2934.12109375 cpu‚25.21692438002647 network.sent‚ 9.019901e+06 disk.dm-1.out‚ 122.9609375! disk.nvme0c0n1.in‚ 18659.46875 disk.nvme0n1.in‚ 18659.46875# disk.dm-0.out‚105497.6064453125% proc.memory.availableMB‚ 1162.5625% disk./.usageGB‚1174.1827430725098KB@¾2Ú- _wandbruntime‚4195  _runtime‚4195iK‚8ÇÛÁ % _timestamp‚1.7884920962702641e+09 _runtime‚4195.910713738 _step‚317 & lr-AdamW/pg1‚0.0006786565779541451 & lr-AdamW/pg2‚0.0006786565779541451 trainer/global_step‚3899³Ýì ÌÜÆ _runtime‚4195.910713738 _step‚317 & lr-AdamW/pg1‚0.0006786565779541451 & lr-AdamW/pg2‚0.0006786565779541451 trainer/global_step‚3899 % _timestamp‚1.7884920962702641e+09½°ƒ*¤2Ý- _wandbruntime‚4195  _runtime‚4195‘˜Cò'ÞìO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68269378 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21770550 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚269665153811104 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚265671555525936 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚276652382546432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚269664366123332 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3992827636756 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4007255006208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚265671538486576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272644354895360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86523150 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _runtime‚4195.917659178 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 _step‚318 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154769512 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86500134 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276651591898624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4008027651072 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68269380 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68269380 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60911852 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226470912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3993598285168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3899 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86523150 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚302755891031040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272644336892416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202530816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4182668401664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016Ç¡ßñO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚269664366123332 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3992827636756 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4007255006208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚265671538486576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272644354895360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86523150 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _runtime‚4195.917659178 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 _step‚318 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154769512 L?DeviceStatsMonitor.on_train_bÁõžÙ atch_start/active.small_pool.freed‚86500134 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276651591898624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4008027651072 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68269380 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68269380 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60911852 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226470912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3993598285168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3899 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86523150 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚302755891031040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272644336892416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202530816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4182668401664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68269378 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21770550 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚269665153811104 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚265671555525936 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚276652382546432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624¾TŒ2à- _wandbruntime‚4197  _runtime‚4197û#Aû&áõM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68286252 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307014764506112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚276721087244800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4009032410112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68286252 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86544887 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60926751 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86521845 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚269732221386950 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3994599425254 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86521845 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 trainer/global_step‚3899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68286254 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154831141 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68286254 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60927292 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚269731433698970 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3993828776634 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4008259751936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154831141 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚302831263313920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276720296583680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307014978380800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21775807 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚319 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4008259751936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272712036831744 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4183715066880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276721087244800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.788492097924725e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86544887 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272712036831744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4009032410112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚276721087244800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4009032410112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68286252 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86544887 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60926751 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86521845 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚269732221386950 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3994599425254 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86521845 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 trainer/global_step‚3899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68286254 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154831141 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68286254 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60927292 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚269731433698970 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3993828776634 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4008259751936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154831141 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚302831263313920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276720296583680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307014978380800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21775807 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚319 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4008259751936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272712036831744 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4183715066880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276721087244800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.788492097924725e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86544887 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272712036831744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4009032410112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307014764506112¿ uŽÆ2ã- _wandbruntime‚4197  _runtime‚4197z¼”‹\äÖ 2loss/SRqqt1/assignment_loss‚1.7414708963849328 2loss/SRqqt3/assignment_loss‚1.1894316056522687 1loss/SRqqt2/detection_loss‚2.8839669710113887 /loss/FBt3/detection_loss‚1.7480856229861577 _step‚320 0loss/FRt4/assignment_loss‚1.5787929672261942 1loss/SRqqt4/detection_loss‚1.0220262597477625 .loss/FBt2/detection_loss‚4.735266307989757 1loss/SRqqt2/assignment_loss‚1.976840852271943 0loss/FBt3/assignment_loss‚0.8116717948578298 /loss/FRt1/detection_loss‚2.3423602441523936 .loss/FRt4/detection_loss‚1.779424183678044 _runtime‚4197.568302939 0loss/FRt1/assignment_loss‚3.1619318989997214 1loss/FBt4/assignment_loss‚0.16566971817599235 /loss/FBt4/detection_loss‚0.5853430197328433 1loss/SRqqt4/assignment_loss‚0.936725180625679 trainer/global_step‚3899 epoch‚3 0loss/FRt2/assignment_loss‚3.7440857387358144 1loss/FBt2/assignment_loss‚0.34414531554405886 /loss/FRt3/detection_loss‚1.9811865833460116 1loss/SRqqt1/detection_loss‚2.7988067106767134 /loss/FBt1/detection_loss‚2.8141785860061646 .loss/FRt2/detection_loss‚2.703916267041237 0loss/FRt3/assignment_loss‚2.3237647647951163 0loss/FBt1/assignment_loss‚0.1067550807361057 &loss/total_loss‚44.844448554179365 1loss/SRqqt3/detection_loss‚1.3686019838052361 $ _timestamp‚1.788492097927964e+09ñ‰aåÛ /loss/FBt4/detection_loss‚0.5853430197328433 1loss/SRqqt4/assignment_loss‚0.936725180625679 trainer/global_step‚3899 epoch‚3 0loss/FRt2/assignment_loss‚3.7440857387358144 1loss/FBt2/assignment_loss‚0.34414531554405886 /loss/FRt3/detection_loss‚1.9811865833460116 1loss/SRqqt1/detection_loss‚2.7988067106767134 /loss/FBt1/detection_loss‚2.8141785860061646 .loss/FRt2/detection_loss‚2.703916267041237 0loss/FRt3/assignment_loss‚2.3237647647951163 0loss/FBt1/assignment_loss‚0.1067550807361057 &loss/total_loss‚44.844448554179365 1loss/SRqqt3/detection_loss‚1.3686019838052361 $ _timestamp‚1.788492097927964e+09 2loss/SRqqt1/assignment_loss‚1.7414708963849328 2loss/SRqqt3/assignment_loss‚1.1894316056522687 1loss/SRqqt2/detection_loss‚2.8839669710113887 /loss/FBt3/detection_loss‚1.7480856229861577 _step‚320 0loss/FRt4/assignment_loss‚1.5787929672261942 1loss/SRqqt4/detection_loss‚1.0220262597477625 .loss/FBt2/detection_loss‚4.735266307989757 1loss/SRqqt2/assignment_loss‚1.976840852271943 0loss/FBt3/assignment_loss‚0.8116717948578298 /loss/FRt1/detection_loss‚2.3423602441523936 .loss/FRt4/detection_loss‚1.779424183678044 _runtime‚4197.568302939 0loss/FRt1/assignment_loss‚3.1619318989997214 1loss/FBt4/assignment_loss‚0.16566971817599235ÀÕ‘¢ø’æ‚(:‡ ÄòèÔÔ–± gpu.0.gpu‚14.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ïû¯¿€ç‚(:õ ÄòèÔØŽš³) proc.memory.availableMB‚ 1064.67578125# proc.memory.rssMB‚ 2934.12109375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.6171875 disk.sda3.out‚ 123.38671875* proc.memory.percent‚17.908453941345215 cpu‚25.216543924872212 proc.cpu.threads‚17 network.sent‚ 9.023844e+06 network.recv‚ 1.413258e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 123.38671875$ memory_percent‚93.50173473358154 disk.sda.out‚ 123.38671875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 18659.46875 disk.nvme0n1.in‚ 18659.46875' disk.nvme0n1.out‚105845.27490234375$ disk.dm-0.out‚105845.27490234375) disk./.usagePercent‚78.80197582723612$ disk./.usageGB‚1174.206386566162 disk.sda1.out‚0) disk.nvme0c0n1.out‚105845.27490234375 disk.dm-0.in‚ 18659.46875 disk.dm-1.in‚0Í Ô6³è‚(:’ ÓòèÔÍÇä¯ gpu.0.gpu‚33.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 ³Åç êgpu.0.powerWatts‚164.398' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.879599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý²›„é‚(:ù Óòèԩdz° disk.sda.in‚0 disk.sda.out‚ 123.69921875 disk.sda1.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚106113.37646484375 network.sent‚ 9.131623e+06 disk.sda3.in‚0 disk.drbd1260.out‚ 0.6171875 disk.sda1.out‚0$ disk.nvme0c0n1.in‚18794.28515625) disk.nvme0c0n1.out‚106113.37646484375" disk.nvme0n1.in‚18794.28515625 disk.dm-0.in‚18794.28515625 network.recv‚ 1.419164e+06 disk.sda2.in‚0$ disk.dm-0.out‚106113.37646484375$ memory_percent‚94.15779113769531$ proc.memory.availableMB‚957.1875) disk./.usagePercent‚78.80145126779846 disk.sda3.out‚ 123.69921875 disk.dm-1.out‚ 123.69921875# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 cpu‚25.21617126817994 proc.cpu.threads‚17% disk./.usageGB‚1174.1985702514648 disk.dm-1.in‚0 disk.drbd1260.in‚0’džl‘ê‚(:† âòèÔ‚“¤¯ gpu.0.gpu‚7.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.769' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.7538 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ÿ½a†ë‚(:û âòèÔÿŒë³) disk.nvme0c0n1.out‚106451.19482421875" disk.nvme0n1.in‚18794.28515625' disk.nvme0n1.out‚106451.19482421875 cpu‚25.215783235939803 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚18794.28515625 disk.dm-0.in‚18794.28515625$ disk.dm-0.out‚106451.19482421875# memory_percent‚94.8225736618042% disk./.usageGB‚1174.2107772827148 disk.dm-1.out‚ 124.1953125 disk.sda.in‚0 disk.sda.out‚ 124.1953125( proc.memory.availableMB‚ 848.26953125# proc.memory.rssMB‚ 2934.12109375 network.recv‚ 1.422795e+06) disk./.usagePercent‚78.80227049190803 disk.sda3.out‚ 124.1953125 disk.drbd1260.out‚ 0.62109375 disk.sda2.in‚0* proc.memory.percent‚17.908453941345215 proc.cpu.threads‚17 network.sent‚ 9.136656e+06 disk.sda3.in‚0 disk.sda1.in‚0ÒA=2ì‚(:’ ñòèÔâàÝ® gpu.0.gpu‚39.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚154.598' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.919600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öOŠzí‚(:ô ñòèÔ²ý«° disk.sda.in‚0# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 cpu‚25.215366103116413$ disk.dm-0.out‚106866.03466796875) disk.nvme0c0n1.out‚106866.03466796875" disk.nvme0n1.in‚18794.28515625' disk.nvme0n1.out‚106866.03466796875" memory_percent‚95.479416847229 proc.cpu.threads‚17 network.sent‚ 9.142909e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.62109375 disk.sda.out‚ 124.71875% disk./.usageGB‚1174.2732162475586 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚18794.28515625 disk.dm-0.in‚18794.28515625 disk.sda3.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 740.65234375 network.recv‚ 1.427392e+06) disk./.usagePercent‚78.80646082322848 disk.sda3.out‚ 124.71875 disk.dm-1.out‚ 124.71875¥bƒž2î- _wandbruntime‚4247  _runtime‚4247o•YÇïÁ _runtime‚4247.669438799 _step‚321 & lr-AdamW/pg1‚0.0006746191859023519 & lr-AdamW/pg2‚0.0006746191859023519 trainer/global_step‚3949 % _timestamp‚1.7884921480291057e+09ó·WŸÌðÆ % _timestamp‚1.7884921480291057e+09 _runtime‚4247.669438799 _step‚321 & lr-AdamW/pg1‚0.0006746191859023519 & lr-AdamW/pg2‚0.0006746191859023519 trainer/global_step‚3949Á=±½2ñ- _wandbruntime‚4247  _runtime‚4247ß»ú©ò'òìO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69113080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276029333858816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚306523308170240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87589036 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 _runtime‚4247.670591732 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87589036 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156725132 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39624235 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4057506706432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚273018546184212 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87612052 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4234896730112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61657396 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69113078 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚322 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚280086840565248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4234683713536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4058279351296 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87612052 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22032608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268974877313936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4043668870276 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61656843 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39624237 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156702114 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276029333858816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4058279351296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69113078 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚273017758496440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22033159 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4042898221864 trainer/global_step‚3949 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚322 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚280086840565248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4234683713536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4058279351296 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87612052 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22032608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268974877313936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4043668870276 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61656843 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39624237 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156702114 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276029333858816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4058279351296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69113078 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚273017758496440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22033159 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4042898221864 trainer/global_step‚3949 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚306598656513024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4043899361742 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69129952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22037797 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚273085613760058 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4058511452160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156740699 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚306598680453120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310834607488512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _step‚323 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280155545250304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276097033798144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61672213 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61671680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156740699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39633885 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 trainer/global_step‚3949 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4058511452160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69129952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚310834378933760 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156763743 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚280156335911424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87610747 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276097051801088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156763743 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87633789 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 % _timestamp‚1.7884921489705174e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22038328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228554752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87633789 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚V8 4059284110336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚280156335911424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39633883 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚269040926710336 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4235722420736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚273084826072078 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4044670010362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4235927035392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276097033798144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚4248.611926429 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276097051801088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚306598656513024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4043899361742 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69129952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22037797 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚273085613760058 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4058511452160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156740699 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚306598680453120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310834607488512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _step‚323 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280155545250304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276097033798144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61672213 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61671680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156740699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39633885 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 trainer/global_step‚3949 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4058511452160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69129952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚310834378933760 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156763743 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚280156335911424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87610747 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276097051801088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156763743 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87633789 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 % _timestamp‚1.7884921489705174e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22038328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228554752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87633789 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4059284110336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚280156335911424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39633883 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚269040926710336 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4235722420736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚273084826072078 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4044670010362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4235927035392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276097033798144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚4248.611926429 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276097051801088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279414348828160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚310290725309440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 179462144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88700954 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4286940926976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4093739455384 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 _runtime‚4298.722249325 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22295025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88700954 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314577486774272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚276371938557320 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88677938 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158634716 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚310290749249536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚272278199101936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279414330825216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4092968806972 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88677938 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283522089231872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4108531051520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283522879879680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283522089231872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4107758406656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62401662 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚272278182062576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69956778 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203402240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4107758406656 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279414348828160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69956780 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22294507 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40106635 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62401142 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40106637 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69956780 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69956778 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚314577690176512 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚314577690176512 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279414348828160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚310290725309440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 179462144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88700954 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4286940926976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4093739455384 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 _runtime‚4298.722249325 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 XKDeviceStatsMonitor.on_train_batch_start/inactiveoè/}_split.small_pool.allocated‚22295025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88700954 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314577486774272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚276371938557320 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88677938 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158634716 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚310290749249536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚272278199101936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279414330825216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4092968806972 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88677938 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283522089231872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4108531051520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283522879879680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283522089231872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4107758406656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62401662 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚272278182062576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69956778 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203402240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4107758406656 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279414348828160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69956780 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22294507 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40106635 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62401142 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40106637 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69956780 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Æ-ië2†- _wandbruntime‚4299  _runtime‚4299ej•žü&‡öM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4109535810560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158673301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62415903 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚314654099656704 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283591584578048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88699649 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4093969946850 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158696345 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283591584578048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88722691 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310366073652224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4288002064384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚272344265537696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22299620 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚283590793916928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚276439006133166 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4108763152384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69973654 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚276438218445186 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158696345 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88722691 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22300160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40116283 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 trainer/global_step‚3999 _step‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88699649 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4109535810560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚279482048767488 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4287801644032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4094740595470 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 % _timestamp‚1.7884921999369888e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62416445 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4108763152384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40116285 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279482030764544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69973652 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4109535810560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158673301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62415903 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚314654099656704 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283591584578048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88699649 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4093969946850 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158696345 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283591584578048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88722691 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310366073652224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4288002064384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚272344265537696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22299620 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚283590793916928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚276439006133166 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4108763152384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69973654 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚276438218445186 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158696345 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88722691 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22300160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40116283 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 trainer/global_step‚3999 _step‚327 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88699649 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4109535810560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚279482048767488 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4287801644032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4094740595470 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 % _timestamp‚1.7884921999369888e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62416445 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4108763152384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40116285 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279482030764544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69973652 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70800480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚330 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89766840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282799327791616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286958128546304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40589035 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286958128546304 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286957337898496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160590336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314058142448640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282799345794560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚275581520889936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4143810040492 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4158010106880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4158010106880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89789856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40589037 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884922491757069e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0•dp¹÷'—ñO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89766840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282799327791616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286958128546304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40589035 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286958128546304 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286957337898496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160590336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314058142448640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282799345794560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚275581520889936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4143810040492 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4158010106880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4158010106880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89789856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40589037 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884922491757069e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70800480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚330 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ÊOÈØ2˜- _wandbruntime‚4349  _runtime‚4349H€ˆ¿ü&™öM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282867045733888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚318473549980160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89811593 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279792398506274 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275647570286336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40598683 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63161900 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314133490791424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4049 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70817354 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40598685 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22563760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22563217 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4159787510784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275647587325696 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160605903 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70817354 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4340259609088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 % _timestamp‚1.7884922501270967e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318473774340608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4144811180578 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282867027730944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚287026042583552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160628947 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160605903 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89811593 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 _step‚331¯À4ª‚šûM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4340259609088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 % _timestamp‚1.7884922501270967e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocatÁ$í$ions.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318473774340608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4144811180578 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282867027730944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚287026042583552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160628947 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160605903 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89811593 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 _step‚331 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282867045733888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚318473549980160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89811593 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279792398506274 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275647570286336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40598683 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63161900 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314133490791424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4049 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70817354 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40598685 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22563760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22563217 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4159787510784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275647587325696 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160605903 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70817354 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Ë;¿’2›- _wandbruntime‚4349  _runtime‚4349AI‹\œÖ 1loss/FBt3/assignment_loss‚0.41483349906694555 /loss/FRt2/detection_loss‚2.7934898206820855 .loss/FRt4/detection_loss‚2.050230960920453 0loss/SRqqt2/detection_loss‚2.161005730288369 1loss/SRqqt3/detection_loss‚1.3833784808197782 0loss/SRqqt4/detection_loss‚1.077225694464425 2loss/SRqqt2/assignment_loss‚1.3620242301906857 1loss/FBt2/assignment_loss‚0.21765607992419972 /loss/FBt2/detection_loss‚2.5395238995552063 /loss/FBt4/detection_loss‚0.5655370749389896 % _timestamp‚1.7884922501293538e+09 _step‚332 0loss/FBt1/assignment_loss‚0.2644405822583477 1loss/FBt4/assignment_loss‚0.10255381574951877 _runtime‚4349.769518926 0loss/FRt3/assignment_loss‚2.5790273944536843 /loss/FRt4/assignment_loss‚1.637677900493145 2loss/SRqqt1/assignment_loss‚1.2757732810621911 /loss/FRt3/detection_loss‚1.9951980136058949 0loss/SRqqt1/detection_loss‚2.793960769068111 /loss/FRt2/assignment_loss‚3.029614895582199 2loss/SRqqt3/assignment_loss‚1.6885071870250006 /loss/FBt3/detection_loss‚2.0077705913119845 /loss/FRt1/detection_loss‚2.7316087282473043 epoch‚3 0loss/FRt1/assignment_loss‚2.8651718055048296 2loss/SRqqt4/assignment_loss‚1.0385870081135782 /loss/FBt1/detection_loss‚2.1081185069951145 %loss/total_loss‚40.68291595032204 trainer/global_step‚4049_³ÎƒaÛ /loss/FRt4/assignment_loss‚1.637677900493145 2loss/SRqqt1/assignment_loss‚1.2757732810621911 /loss/FRt3/detection_loss‚1.9951980136058949 0loss/SRqqt1/detection_loss‚2.793960769068111 /loss/FRt2/assignment_loss‚3.029614895582199 2loss/SRqqt3/assignment_loss‚1.6885071870250006 /loss/FBt3/detection_loss‚2.0077705913119845 /loss/FRt1/detection_loss‚2.7316087282473043 epoch‚3 0loss/FRt1/assignment_loss‚2.8651718055048296 2loss/SRqqt4/assignment_loss‚1.0385870081135782 /loss/FBt1/detection_loss‚2.1081185069951145 %loss/total_loss‚40.68291595032204 trainer/global_step‚4049 1loss/FBt3/assignment_loss‚0.41483349906694555 /loss/FRt2/detection_loss‚2.7934898206820855 .loss/FRt4/detection_loss‚2.050230960920453 0loss/SRqqt2/detection_loss‚2.161005730288369 1loss/SRqqt3/detection_loss‚1.3833784808197782 0loss/SRqqt4/detection_loss‚1.077225694464425 2loss/SRqqt2/assignment_loss‚1.3620242301906857 1loss/FBt2/assignment_loss‚0.21765607992419972 /loss/FBt2/detection_loss‚2.5395238995552063 /loss/FBt4/detection_loss‚0.5655370749389896 % _timestamp‚1.7884922501293538e+09 _step‚332 0loss/FBt1/assignment_loss‚0.2644405822583477 1loss/FBt4/assignment_loss‚0.10255381574951877 _runtime‚4349.769518926 0loss/FRt3/assignment_loss‚2.5790273944536843ÌAm$ž‚(:’ ÚóèÔÁÜÙ± gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.952' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.590399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªb †Ÿ‚(:û ÚóèÔ»Ÿ¬²) disk.nvme0c0n1.out‚109396.34228515625$ memory_percent‚96.56705856323242& proc.memory.availableMB‚ 562.453125% disk./.usageGB‚1175.2112922668457 disk.drbd1260.in‚0" disk.nvme0n1.in‚19069.57421875# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 proc.cpu.threads‚17 network.sent‚ 9.398932e+06 disk.sda.out‚ 129.26171875 disk.sda1.in‚0 disk.sda2.in‚0$ disk.dm-0.out‚109396.34228515625 cpu‚25.212833573517447 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚19069.57421875' disk.nvme0n1.out‚109396.34228515625 disk.dm-0.in‚19069.57421875 network.recv‚ 1.459151e+06) disk./.usagePercent‚78.86941589198105 disk.sda3.out‚ 129.26171875 disk.dm-1.out‚ 129.26171875 disk.drbd1260.out‚0.640625’Æ  ‚(:’ éóèÔ¶ƒÄ® gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚128.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.791000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y_€¡‚(:õ éóèÔ”Œæ±% disk./.usageGB‚1175.1936950683594" disk.nvme0n1.in‚19069.57421875& disk.nvme0n1.out‚109823.6064453125 disk.dm-0.in‚19069.57421875 disk.sda2.in‚0# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215$ disk.nvme0c0n1.in‚19069.57421875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.640625 disk.sda.out‚ 129.8046875 disk.sda1.out‚0$ memory_percent‚96.56586647033691 cpu‚25.212515969412422 disk.sda3.in‚0 disk.sda3.out‚ 129.8046875 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.465805e+06( disk./.usagePercent‚78.8682349292256( disk.nvme0c0n1.out‚109823.6064453125# disk.dm-0.out‚109823.6064453125 disk.dm-1.out‚ 129.8046875 disk.sda.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 562.6484375 network.sent‚ 9.495111e+06nxs’¢‚(:‡ øóèÔÔÁ£¯ gpu.0.gpu‚19.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø²±àv£‚(:ë øóèÔ—ÔÛ± network.sent‚ 9.500879e+06 disk.dm-1.out‚ 130.234375 disk.drbd1260.out‚ 0.64453125 disk.sda.out‚ 130.234375% disk.nvme0c0n1.out‚110261.8203125# disk.nvme0n1.out‚110261.8203125 cpu‚25.212137981004698 proc.cpu.threads‚17 disk.dm-0.out‚110261.8203125 disk.sda1.out‚0$ memory_percent‚96.56744003295898& proc.memory.availableMB‚ 562.390625 network.recv‚ 1.470221e+06) disk./.usagePercent‚78.86980988757624 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚19069.57421875" disk.nvme0n1.in‚19069.57421875* proc.memory.percent‚17.908453941345215% disk./.usageGB‚1175.2171630859375 disk.dm-0.in‚19069.57421875 disk.sda3.in‚0 disk.sda3.out‚ 130.234375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2934.12109375ô­u†’¤‚(:‡ ‡ôèÔ–¤Ç® gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚126.547' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.3094 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿÏç˜j¥‚(:ß ‡ôèÔÈÔÚ²$ memory_percent‚96.56586647033691 cpu‚25.211751333800724 network.recv‚ 1.474013e+06 disk.sda.out‚130.5625 disk.sda1.in‚0" disk.nvme0n1.in‚19069.57421875# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 proc.cpu.threads‚17% disk./.usageGB‚1175.1936950683594 disk.sda3.out‚130.5625 disk.dm-1.in‚0 disk.dm-1.out‚130.5625 disk.drbd1260.in‚0 disk.sda2.in‚0! disk.nvme0n1.out‚ 110652.59375 disk.dm-0.in‚19069.57421875 network.sent‚ 9.506033e+06 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚19069.57421875# disk.nvme0c0n1.out‚ 110652.59375 disk.dm-0.out‚ 110652.59375' proc.memory.availableMB‚ 562.6484375( disk./.usagePercent‚78.8682349292256 disk.drbd1260.out‚ 0.64453125 disk.sda1.out‚0aÙƒ2¦- _wandbruntime‚4399  _runtime‚4399-…ŧ¿ _step‚333 & lr-AdamW/pg1‚0.0006621622040312611 & lr-AdamW/pg2‚0.0006621622040312611 trainer/global_step‚4099 % _timestamp‚1.7884922998117115e+09 _runtime‚ 4399.45207144éé欀 & lr-AdamW/pg1‚0.0006621622040312611 & lr-AdamW/pg2‚0.0006621622040312611 trainer/global_step‚4099 % _timestamp‚1.7884922998117115e+09 _runtime‚ 4399.4520714 _step‚333Í„u‚82©- _wandbruntime‚4399  _runtime‚4399î¯ò'ªìO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71644180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41071435 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4391209041920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278884825638576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162522938 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71644178 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚283077935615764 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90855742 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286184324758016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290392586565120 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚322216768629760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4193109977188 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚290393377212928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286184342760960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90855742 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4193880625600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4209034451968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317825583527936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71644180 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328p$YÒë«ñO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚283078723303536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4209034451968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22821955 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41071437 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚290393377212928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71644178 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286184324758016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290392586565120 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚322216768629760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4193109977188 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚290393377212928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286184342760960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90855742 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4193880625600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4209034451968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317825583527936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71644180 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71644180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41071435 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4391209041920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278884825638576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162522938 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71644178 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚283077935615764 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90855742 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 4400.48046788 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63908311 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322293169430528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90900495 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90877453 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63907767 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4392261499904 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚290462081911296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4209266552832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚290461291250176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚283145790879382 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884923008393066e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286252042700288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4210039211008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162561549 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 trainer/global_step‚4099 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90900495 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162561549 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71661054 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22826684 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328ZtzÓ'®úM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71661054 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322293383305216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚317900931870720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4194111117066 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162538505 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4194881765686 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚286252024697344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚290462081911296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4209266552832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚290461291250176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚283145790879382 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884923008393066e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286252042700288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4210039211008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162561549 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 trainer/global_step‚4099 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90900495 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162561549 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71661054 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22826684 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290462081911296 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286252024697344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 4400.48046788 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63908311 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322293169430528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90900495 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90877453 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63907767 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4392261499904 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72487880 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚293828625879552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41553835 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72487878 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72487878 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚293827835231744 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _runtime‚4450.984494462 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚282188147426576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4243951210708 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289569339727360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326036769249792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226470912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91944644 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4149 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚282188164465936 % _timestamp‚1.7884923513437757e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4258513507328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91967660 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚321593000667136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91944644 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64640605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23086232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4243180562296 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293827835231744 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289569321724416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4443768582656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64640067 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825±ø€÷'½ñO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72487878 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72487878 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚293827835231744 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _runtime‚4450.984494462 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚282188147426576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4243951210708 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289569339727360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326036769249792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226470912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91944644 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4149 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚282188164465936 % _timestamp‚1.7884923513437757e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4258513507328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91967660 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚321593000667136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91944644 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64640605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23086232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4243180562296 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293827835231744 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289569321724416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4443768582656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64640067 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4259286152192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289569339727360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289569321724416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4259286152192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326036542778880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23086768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4443566051840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72487880 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚293828625879552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41553835 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489Òö@ßä2¾- _wandbruntime‚4452  _runtime‚4452 tëü&¿öM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _step‚339 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326113162526208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚282254213862336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289637039666688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321668325069824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293897330577920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4452.179878653 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164494151 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289637021663744 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64655007 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚321668349009920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72504754 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41563485 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41563483 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4244952350794 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚293897330577920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64655531 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72504752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4260290911232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164494151 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 trainer/global_step‚4149 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164471107 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326112944457216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚282254230901696ú"ˆÊ ÀûM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4444813516288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72504754 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286498395564510 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚289637021663744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4444619387392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 % _timestamp‚1.7884923525381129e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91989397 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194128896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286499183252490 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164494151 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ZFDeviceStatsMonitor.on_train_bat~°,0öch_end/allocated_bytes.large_pool.freed‚289637021663744 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64655007 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚321668349009920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72504754 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41563485 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41563483 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4244952350794 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚293897330577920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64655531 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72504752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4260290911232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164494151 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 trainer/global_step‚4149 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164471107 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326112944457216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚282254230901696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _step‚339 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326113162526208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚282254213862336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289637039666688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321668325069824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293897330577920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4452.179878653 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 BÈ¥’Ê‚(:‡ ðôèԤ췭 gpu.0.gpu‚44.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01Å ó}Ë‚(:ò ðôèÔÈƇ¯ disk.sda2.out‚0$ memory_percent‚96.56660556793213 cpu‚25.20933981529765' disk.nvme0c0n1.out‚113517.533203125 disk.drbd1260.in‚0 disk.sda.out‚ 133.796875 disk.sda1.in‚0( proc.memory.availableMB‚ 562.52734375# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 133.796875 network.recv‚ 1.507878e+06) disk./.usagePercent‚78.79324622632606% disk./.usageGB‚1174.0763092041016$ disk.nvme0c0n1.in‚19352.80859375" disk.nvme0n1.in‚19352.80859375% disk.nvme0n1.out‚113517.533203125 disk.sda3.out‚ 133.796875 disk.dm-0.in‚19352.80859375 disk.drbd1260.out‚ 0.66015625 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 9.751656e+06" disk.dm-0.out‚113517.533203125÷|}º2Ì- _wandbruntime‚4502  _runtime‚4502A´LÆÍÀ _step‚341 & lr-AdamW/pg1‚0.0006535788940517993 & lr-AdamW/pg2‚0.0006535788940517993 trainer/global_step‚4199 $ _timestamp‚1.788492402474114e+09 _runtime‚4502.114507459¾B»<ËÎÅ & lr-AdamW/pg1‚0.0006535788940517993 & lr-AdamW/pg2‚0.0006535788940517993 trainer/global_step‚4199 $ _timestamp‚1.788492402474114e+09 _runtime‚4502.114507459 _step‚341Õƒ äð2Ï- _wandbruntime‚4502  _runtime‚4502'Båò'ÐìO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚329856426491904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4308765207552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4308765207552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚329856210506752 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73331578 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23351138 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292954336693760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73331580 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4309537852416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚289785508049752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65387905 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4309537852416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚325360417806336 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73331580 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166388142 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42036235 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚4502.115577536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚292954336693760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285491486253936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285491469214576 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73331578 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297263083898368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42036237 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7884924024751449e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚292954318690816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4293251147404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93033546 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚342 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4496008685568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚289784720361980 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 trainer/global_step‚4199 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚297263083898368 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297263083898368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42036237 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7884924024751449e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚292954318690816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4293251147404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93033546 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚342 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4496008685568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚289784720361980 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 trainer/global_step‚4199 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚297263083898368 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73331578 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23351138 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeA.^zviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292954336693760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73331580 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4309537852416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚289785508049752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65387905 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4309537852416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚325360417806336 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73331580 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166388142 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42036235 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚4502.115577536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚292954336693760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285491486253936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285491469214576 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73331578 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884924033266428e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297331788583424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4496862610432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93055257 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23356911 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166403709 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4497058836480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4295022935902 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297332579244544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42045883 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚293022036633088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23356391 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4502.967636254 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285557552689696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166426753 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚343 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65402274 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166403709 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚293022036633088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚289851787937618 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516„^U8'ÔûM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4310542611456 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297332579244544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65402274 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166403709 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚293022036633088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚289851787937618 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4309769953280 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884924033266428e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297331788583424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4496862610432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93055257 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23356911 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166403709 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4497058836480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4295022935902 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297332579244544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42045883 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚293022036633088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23356391 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4502.967636254 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285557552689696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166426753 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚343 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4359016907776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74175278 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296339315657216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4548213291520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42518637 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168320744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23615751 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296339333660160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4359016907776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333676024296960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296339315657216 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94145464 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74175278 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4343321732512 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23616278 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74175280 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329127811005440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66134915 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300698332564992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94122448 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94122448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 trainer/global_step‚4249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300698332564992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 % _timestamp‚1.7884924542850578e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66134915 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300698332564992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94122448 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94122448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 trainer/global_step‚4249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300698332564992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 % _timestamp‚1.7884924542850578e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4359016907776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74175278 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296339315657216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4548213291520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42518637 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168320744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23615751 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296339333660160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4359016907776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333676024296960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328ÚÒä€l2ä- _wandbruntime‚4554  _runtime‚4554sáäŸü&åöM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296407015596544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4360021653504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66149829 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚288860874477696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74192152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329203183288320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4360794311680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74192154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4549264130560 _step‚347 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296407015596544 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4344322872390 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168359355 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚4249 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 % _timestamp‚1.7884924551457918e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168359355 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4549456162304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23621028 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333752423478784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚293205180310726 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74192154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚296407033599488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚333752639450624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4345093521010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300767827911168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42528285 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66149311 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42528283 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94167201 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288860857438336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4360021653504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚04ÁÿØ– æûM EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168359355 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚4249 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 % _timestamp‚1.7884924551457918e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168359355 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4549456162304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23621028 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333752423478784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚293205180310726 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74192154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚296407033599488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚333752639450624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4345093521010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300767827911168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42528285 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66149311 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42528283 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94167201 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288860857438336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4360021653504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296407015596544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4360021653504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66149829 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚288860874477696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74192152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329203183288320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4360794311680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74192154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4549264130560 _step‚347 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296407015596544 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4344322872390 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075Û¦“&2ç- _wandbruntime‚4554  _runtime‚4554y¨L]è× trainer/global_step‚4249 _runtime‚4554.788046567 0loss/FRt2/assignment_loss‚3.2129862026164404 0loss/FBt1/assignment_loss‚0.1947286080030608 /loss/FBt1/detection_loss‚3.6512171551585197 1loss/FBt4/assignment_loss‚0.13038353337577513 0loss/FRt1/assignment_loss‚3.6175586617722804 /loss/FRt3/assignment_loss‚2.748131429895442 2loss/SRqqt4/assignment_loss‚0.8412627440226811 2loss/SRqqt3/assignment_loss‚1.4209369509606748 1loss/FBt3/assignment_loss‚0.17802325334050692 .loss/FRt3/detection_loss‚2.259067559178839 1loss/SRqqt3/detection_loss‚1.3256648467803323 /loss/FBt3/detection_loss‚2.4953581154346467 /loss/FBt4/detection_loss‚0.5297337773525598 0loss/FBt2/assignment_loss‚0.4302290206942416 .loss/FRt2/detection_loss‚2.277423615588389 epoch‚3 % _timestamp‚1.7884924551478922e+09 0loss/FRt4/assignment_loss‚1.6665510276578508 2loss/SRqqt1/assignment_loss‚1.4170766101617898 /loss/FRt1/detection_loss‚2.0939482021696714 /loss/FRt4/detection_loss‚1.6738304564694189 0loss/SRqqt2/detection_loss‚2.167352353675025 1loss/SRqqt4/detection_loss‚1.0076515574067357 /loss/FBt2/detection_loss‚2.3276289020265852 2loss/SRqqt2/assignment_loss‚1.7461418867111207 &loss/total_loss‚41.925652278217456 _step‚348 0loss/SRqqt1/detection_loss‚2.512765807764871él …béÜ 0loss/FRt2/assignment_loss‚3.2129862026164404 0loss/FBt1/assignment_loss‚0.1947286080030608 /loss/FBt1/detection_loss‚3.6512171551585197 1loss/FBt4/assignment_loss‚0.13038353337577513 0loss/FRt1/assignment_loss‚3.6175586617722804 /loss/FRt3/assignment_loss‚2.748131429895442 2loss/SRqqt4/assignment_loss‚0.8412627440226811 2loss/SRqqt3/assignment_loss‚1.4209369509606748 1loss/FBt3/assignment_loss‚0.17802325334050692 .loss/FRt3/detection_loss‚2.259067559178839 1loss/SRqqt3/detection_loss‚1.3256648467803323 /loss/FBt3/detection_loss‚2.4953581154346467 /loss/FBt4/detection_loss‚0.5297337773525598 0loss/FBt2/assignment_loss‚0.4302290206942416 .loss/FRt2/detection_loss‚2.277423615588389 epoch‚3 % _timestamp‚1.7884924551478922e+09 0loss/FRt4/assignment_loss‚1.6665510276578508 2loss/SRqqt1/assignment_loss‚1.4170766101617898 /loss/FRt1/detection_loss‚2.0939482021696714 /loss/FRt4/detection_loss‚1.6738304564694189 0loss/SRqqt2/detection_loss‚2.167352353675025 1loss/SRqqt4/detection_loss‚1.0076515574067357 /loss/FBt2/detection_loss‚2.3276289020265852 2loss/SRqqt2/assignment_loss‚1.7461418867111207 &loss/total_loss‚41.925652278217456 _step‚348 0loss/SRqqt1/detection_loss‚2.512765807764871 trainer/global_step‚4249 _runtime‚4554.788046567Ü’íÁ‘ê‚(:† ¬õèÔ†òô® gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.352' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.8704 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ÈJë‚(:ö ¬õèÔàµò¯* proc.memory.percent‚17.908453941345215 disk.drbd1260.in‚0 disk.sda.in‚0" disk.dm-0.out‚114769.404296875( proc.memory.availableMB‚ 561.68359375# proc.memory.rssMB‚ 2934.12109375 proc.cpu.threads‚17 network.sent‚ 9.876684e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 135.4765625 disk.sda2.in‚0$ disk.nvme0c0n1.in‚19482.26171875' disk.nvme0c0n1.out‚114769.404296875% disk.nvme0n1.out‚114769.404296875 disk.dm-0.in‚19482.26171875 network.recv‚ 1.525964e+06% disk./.usageGB‚1172.9475173950195 disk.drbd1260.out‚ 0.66796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.in‚19482.26171875 cpu‚25.207916563562698) disk./.usagePercent‚78.71749206089915 disk.sda3.in‚0 disk.sda3.out‚ 135.4765625 disk.sda.out‚ 135.4765625$ memory_percent‚96.57175540924072¬ˆÝ³’ì‚(:‡ »õèÔèή gpu.0.gpu‚39.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚130.682' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.1364 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ü Ž~í‚(:ó »õèÔ¬•í² disk.sda1.in‚0' disk.nvme0c0n1.out‚115358.580078125" disk.nvme0n1.in‚19482.26171875( proc.memory.availableMB‚ 562.10546875 network.sent‚ 9.984895e+06 disk.sda3.out‚ 135.984375 disk.drbd1260.out‚ 0.66796875 disk.sda.in‚0% disk.nvme0n1.out‚115358.580078125 disk.dm-0.in‚19482.26171875$ memory_percent‚96.56918048858643# proc.memory.rssMB‚ 2934.12109375) disk./.usagePercent‚78.72643594011537% disk./.usageGB‚1173.0807876586914 disk.sda3.in‚0 disk.sda.out‚ 135.984375 disk.sda2.out‚0$ disk.nvme0c0n1.in‚19482.26171875* proc.memory.percent‚17.908453941345215 cpu‚25.207535693522217 disk.dm-1.in‚0 disk.sda2.in‚0" disk.dm-0.out‚115358.580078125 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚ 1.531504e+06 disk.dm-1.out‚ 135.984375 disk.drbd1260.in‚0[Ësû„î‚(:ù ÊõèÔÍÐ¥² disk.sda1.out‚0$ disk.nvme0c0n1.in‚19482.26171875( disk.nvme0c0n1.out‚115526.3798828125" disk.nvme0n1.in‚19482.26171875# disk.dm-0.out‚115526.3798828125 disk.drbd1260.out‚0.671875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚115526.3798828125$ memory_percent‚96.57065868377686 cpu‚25.207143362604526 network.sent‚ 9.991736e+06) disk./.usagePercent‚78.72830914724341$ disk./.usageGB‚1173.108699798584 disk.sda3.in‚0 disk.sda3.out‚ 136.44921875 disk.drbd1260.in‚0( proc.memory.availableMB‚ 561.86328125# proc.memory.rssMB‚ 2934.12109375 disk.dm-1.in‚0 disk.dm-1.out‚ 136.44921875 disk.sda.out‚ 136.44921875 disk.dm-0.in‚19482.26171875* proc.memory.percent‚17.908453941345215 proc.cpu.threads‚17 network.recv‚ 1.534902e+06 disk.sda.in‚0?¹“è’ï‚(:‡ ÊõèÔæÆÆÆ gpu.0.gpu‚18.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8176 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á&&2ð- _wandbruntime‚4600  _runtime‚4600±œýÇñÁ & lr-AdamW/pg1‚0.0006447809799419009 & lr-AdamW/pg2‚0.0006447809799419009 trainer/global_step‚4299 % _timestamp‚1.7884925006435301e+09 _runtime‚4600.283908931 _step‚349rνÌòÆ _runtime‚4600.283908931 _step‚349 & lr-AdamW/pg1‚0.0006447809799419009 & lr-AdamW/pg2‚0.0006447809799419009 trainer/global_step‚4299 % _timestamp‚1.7884925006435301e+09Ým¶¿l2ó- _wandbruntime‚4600  _runtime‚4600nhÄmò'ôìO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚292098129829936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4410041252864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43001037 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304134371879424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170253346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170230328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚299724330626560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚332895252084736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚299724312623616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4600675641344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95211350 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4410041252864 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4394162966032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95234366 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304134371879424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337495927726080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚332895228144640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4393392317620 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304133581231616 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 % _timestamp‚1.7884925006446147e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226470912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚296492292795968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4409268608000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66882048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66881494 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75018978 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 _step‚350 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚337495701255168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4600473110528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚292098112790576 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304133581231616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299724330626560 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299724312623616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75018980 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4409268608000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489’¶4„"õñO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66882048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66881494 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75018978 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 _step‚350 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚337495701255168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4600473110528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚292098112790576 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304133581231616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299724330626560 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299724312623616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75018980 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4409268608000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚292098129829936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4410041252864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43001037 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304134371879424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170253346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170230328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚299724330626560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚332895252084736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚299724312623616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4600675641344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95211350 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4410041252864 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4394162966032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95234366 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304134371879424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337495927726080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚332895228144640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4393392317620 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1QóÀžä017634825 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304133581231616 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 % _timestamp‚1.7884925006446147e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226470912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚296492292795968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4409268608000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4299Þ!$t¡2ö- _wandbruntime‚4601  _runtime‚4601Ã.–^ü&÷öM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66896378 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4601516713472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43010683 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4410273353728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292164179226336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4411046011904 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚304203076577792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304202285916672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95256103 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43010685 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299792030565888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75035852 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95233061 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170268913 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170291957 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296558572683834 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4410273353728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337572317561344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4601717133824 % _timestamp‚1.7884925014923499e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75035854 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304203076577792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75035854 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299792012562944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4394393457498 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516¢¡à 'øûM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23885695 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66896378 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4601516713472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43010683 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4410273353728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292164179226336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4411046011904 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚304203076577792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304202285916672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95256103 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43010685 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299792030565888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75035852 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95233061 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170268913 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170291957 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296558572683834 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4410273353728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337572317561344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4601717133824 % _timestamp‚1.7884925014923499e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75035854 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304203076577792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75035854 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299792012562944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4394393457498 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 P¼Ç’ü‚(:‡ ÙõèÔŠùì­ gpu.0.gpu‚17.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.727' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q„†~ý‚(:ó ÙõèÔãèÁ²' proc.memory.availableMB‚ 561.3671875# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 disk.sda.in‚0 disk.dm-1.in‚0 cpu‚25.206744026677644 proc.cpu.threads‚17 network.sent‚ 9.996016e+06 network.recv‚ 1.539432e+06% disk./.usageGB‚1173.1096153259277 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 19594.265625# disk.dm-0.out‚115626.2158203125 disk.dm-1.out‚ 136.89453125) disk./.usagePercent‚78.72837058905162 disk.sda.out‚ 136.89453125 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 19594.265625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.671875$ memory_percent‚96.57368659973145" disk.nvme0c0n1.in‚ 19594.265625( disk.nvme0c0n1.out‚115626.2158203125& disk.nvme0n1.out‚115626.2158203125 disk.sda3.in‚0 disk.sda3.out‚ 136.89453125?,à“’þ‚(:‡ èõèÔ´„Õ­ gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.862' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3724 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!¡’~ÿ‚(:ó èõèÔ€÷â° disk.drbd1260.out‚ 0.67578125* proc.memory.percent‚17.908453941345215 proc.cpu.threads‚17 disk.sda.out‚ 137.45703125 disk.sda3.in‚0 disk.dm-1.out‚ 137.45703125# proc.memory.rssMB‚ 2934.12109375 cpu‚25.20636975263829 network.recv‚ 1.545735e+06 disk.sda2.in‚0 disk.nvme0n1.in‚ 19594.265625 disk.sda3.out‚ 137.45703125 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚96.57082557678223' proc.memory.availableMB‚ 561.8359375 network.sent‚ 1.0104819e+07$ disk./.usageGB‚1173.172134399414 disk.sda1.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 19594.265625( disk.nvme0c0n1.out‚115896.2294921875& disk.nvme0n1.out‚115896.2294921875# disk.dm-0.out‚115896.2294921875( disk./.usagePercent‚78.7325662965303 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 19594.265625u4¥’€‚(:‡ ÷õèÔà”ý® gpu.0.gpu‚46.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.573' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5146 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 žh{‚(:ð ÷õèÔº“ö² network.sent‚ 1.0109173e+07) disk./.usagePercent‚78.74360790148206 disk.sda3.out‚ 137.828125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.67578125 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 19594.265625 disk.sda3.in‚0 disk.dm-1.out‚ 137.828125 disk.sda.in‚0 disk.nvme0n1.in‚ 19594.265625& disk.nvme0n1.out‚116406.2412109375 disk.dm-0.in‚ 19594.265625# disk.dm-0.out‚116406.2412109375* proc.memory.percent‚17.908453941345215 disk.sda.out‚ 137.828125 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚116406.2412109375( proc.memory.availableMB‚ 561.83203125 cpu‚25.20600757856925 proc.cpu.threads‚17 network.recv‚ 1.549059e+06% disk./.usageGB‚1173.3366622924805 disk.sda2.in‚0$ memory_percent‚96.57084941864014# proc.memory.rssMB‚ 2934.12109375lóF2‚- _wandbruntime‚4648  _runtime‚4648†BíǃÁ _runtime‚4648.302593386 _step‚353 & lr-AdamW/pg1‚0.0006403038312372947 & lr-AdamW/pg2‚0.0006403038312372947 trainer/global_step‚4349 % _timestamp‚1.7884925486621556e+09|s CÌ„Æ & lr-AdamW/pg2‚0.0006403038312372947 trainer/global_step‚4349 % _timestamp‚1.7884925486621556e+09 _runtime‚4648.302593386 _step‚353 & lr-AdamW/pg1‚0.0006403038312372947á—2t\2…- _wandbruntime‚4648  _runtime‚4648#i/“`†ìO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚299845685169076 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75862680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96300252 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚295401451617936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4444233551140 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚303109309590016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚569 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295401434578576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24146395 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75862678 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚303109327592960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341315788206592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4652916451840 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226470912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336662645283840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚4349 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚303109327592960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocatio{E9#’ns.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4460292953088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884925486633756e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚307569620546048 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43483437 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336662669223936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚299844897481304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75862680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43483435 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24145828 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚567 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202530816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚307569620546048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96323268 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67629263 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4653118982656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4443462902728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4459520308224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96323268 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67629832 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307568829898240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4459520308224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172162930 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4460292953088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75862678 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚307568829898240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚303109309590016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _step‚354 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚307569620546048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96323268 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67629263 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4653118982656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4443462902728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4459520308224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96323268 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67629832 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307568829898240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4459520308224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172162930 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4460292953088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75862678 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚307568829898240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚303109309590016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _step‚354 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚303109309590016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚569 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295401434578576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24146395 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75862678 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚303109327592960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341315788206592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4652916451840 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226470912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336662645283840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0âÅÍX02ˆ- _wandbruntime‚4649  _runtime‚4649´æ¨Vü&‰öM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4460525053952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚303177009529344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4445234691226 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚307637534583296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75879554 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96345005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚295467501014336 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172201515 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 _runtime‚4649.304962179 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96345005 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚303177027532288 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 trainer/global_step‚4349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96321963 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341392178304512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚303177027532288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4654160737792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚303177009529344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4461297712128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24151656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172224559 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67644191 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4653964511744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4444464042606 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336737993626624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚307638325244416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172201515 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67644741 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43493083 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚295467518053696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚299911965056942 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172224559 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75879552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43493085 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328‰!t™&ŠûM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4460525053952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚303177009529344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4445234691226 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚307637534583296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75879554 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96345005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚295467501014336 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172201515 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 _runtime‚4649.304962179 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96345005 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚303177027532288 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 trainer/global_step‚4349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96321963 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341392178304512 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚303177027532288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4654160737792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚303177009529344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4461297712128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24151656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172224559 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67644191 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4653964511744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4444464042606 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336737993626624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚307638325244416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172201515 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67644741 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚êÛ43493083 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚295467518053696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚299911965056942 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172224559 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75879552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43493085 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576㱺Áz2‹- _wandbruntime‚4649  _runtime‚4649Oñ-RZŒÔ 2loss/SRqqt1/assignment_loss‚1.2946069430825966 /loss/FBt4/assignment_loss‚0.116491581226713 epoch‚3 0loss/FBt3/assignment_loss‚0.4993914846563712 /loss/FRt3/detection_loss‚2.1215779228643936 1loss/SRqqt4/detection_loss‚0.9099375174218227 .loss/FBt2/detection_loss‚3.084669311841329 2loss/SRqqt3/assignment_loss‚1.6269351995335175 .loss/FRt4/detection_loss‚1.536573118219773 .loss/FBt1/detection_loss‚3.103519363535775 /loss/FBt4/detection_loss‚0.7729855668048063 0loss/FRt4/assignment_loss‚1.5074178532317832 2loss/SRqqt4/assignment_loss‚0.8819922991096973 2loss/FBt2/assignment_loss‚0.024553646871178596 /loss/FRt1/detection_loss‚2.3849693249378885 1loss/SRqqt3/detection_loss‚1.3646599722596315 $loss/total_loss‚41.4333907759526 trainer/global_step‚4349 0loss/FRt1/assignment_loss‚3.0945444901784263 $ _timestamp‚1.788492549666455e+09 0loss/FRt3/assignment_loss‚2.6207708727229724 /loss/FRt2/detection_loss‚2.6026322416779117 .loss/FBt3/detection_loss‚2.100173090185438 2loss/SRqqt2/assignment_loss‚1.3220758363604546 _runtime‚4649.306671892 _step‚356 /loss/FRt2/assignment_loss‚3.524702842746462 1loss/FBt1/assignment_loss‚0.35803650815753624 1loss/SRqqt1/detection_loss‚2.3054316150290624 1loss/SRqqt2/detection_loss‚2.2747421732970645*,q_Ù 2loss/SRqqt2/assignment_loss‚1.3220758363604546 _runtime‚4649.306671892 _step‚356 /loss/FRt2/assignment_loss‚3.524702842746462 1loss/FBt1/assignment_loss‚0.35803650815753624 1loss/SRqqt1/detection_loss‚2.3054316150290624 1loss/SRqqt2/detection_loss‚2.2747421732970645 2loss/SRqqt1/assignment_loss‚1.2946069430825966 /loss/FBt4/assignment_loss‚0.116491581226713 epoch‚3 0loss/FBt3/assignment_loss‚0.4993914846563712 /loss/FRt3/detection_loss‚2.1215779228643936 1loss/SRqqt4/detection_loss‚0.9099375174218227 .loss/FBt2/detection_loss‚3.084669311841329 2loss/SRqqt3/assignment_loss‚1.6269351995335175 .loss/FRt4/detection_loss‚1.536573118219773 .loss/FBt1/detection_loss‚3.103519363535775 /loss/FBt4/detection_loss‚0.7729855668048063 0loss/FRt4/assignment_loss‚1.5074178532317832 2loss/SRqqt4/assignment_loss‚0.8819922991096973 2loss/FBt2/assignment_loss‚0.024553646871178596 /loss/FRt1/detection_loss‚2.3849693249378885 1loss/SRqqt3/detection_loss‚1.3646599722596315 $loss/total_loss‚41.4333907759526 trainer/global_step‚4349 0loss/FRt1/assignment_loss‚3.0945444901784263 $ _timestamp‚1.788492549666455e+09 0loss/FRt3/assignment_loss‚2.6207708727229724 /loss/FRt2/detection_loss‚2.6026322416779117 .loss/FBt3/detection_loss‚2.100173090185438ä)ÏO¬’Ž‚(:‡ †öèÔÓ„Ü® gpu.0.gpu‚39.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.856' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.1712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i¦ç€‚(:õ †öèÔõóþ° disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.dm-0.out‚116610.1630859375 disk.sda3.out‚ 138.21484375 network.recv‚ 1.552836e+06) disk./.usagePercent‚78.74514445870258 disk.dm-1.out‚ 138.21484375 disk.sda.out‚ 138.21484375 disk.nvme0n1.in‚ 19594.265625 disk.dm-0.in‚ 19594.265625 disk.sda3.in‚0* proc.memory.percent‚17.908453941345215% disk./.usageGB‚1173.3595581054688 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚116610.1630859375$ memory_percent‚96.57111167907715# proc.memory.rssMB‚ 2934.12109375 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 19594.265625( disk.nvme0c0n1.out‚116610.1630859375' proc.memory.availableMB‚ 561.7890625 cpu‚25.205639770008037 network.sent‚ 1.0114205e+07 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.6796875 w™’‚(:‡ •öèÔš¶™¶ gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.261' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6522 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0̾ãU…‘‚(:ú •öèÔðÕ˺& disk.nvme0n1.out‚116927.9892578125 disk.dm-0.in‚19740.80859375 disk.sda3.in‚0 disk.sda1.in‚0 network.sent‚ 1.0225152e+07 network.recv‚ 1.559796e+06 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 562.015625 proc.cpu.threads‚17) disk./.usagePercent‚78.76192012040629% disk./.usageGB‚1173.6095275878906 disk.sda3.out‚ 138.67578125 disk.sda2.out‚0 cpu‚25.205357041385206( disk.nvme0c0n1.out‚116927.9892578125# disk.dm-0.out‚116927.9892578125 disk.drbd1260.out‚ 0.6796875 disk.sda.out‚ 138.67578125 disk.sda2.in‚0$ memory_percent‚96.56972885131836# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 disk.dm-1.in‚0 disk.dm-1.out‚ 138.67578125$ disk.nvme0c0n1.in‚19740.80859375" disk.nvme0n1.in‚19740.80859375ÇB”¤’’‚(:‡ ¤öèÔžÜö± gpu.0.gpu‚41.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gl…“‚(:ú ¤öèÔ¤¡¶ disk.sda1.out‚0' disk.nvme0n1.out‚117126.04248046875' proc.memory.availableMB‚ 562.0078125 network.sent‚ 1.022899e+07 network.recv‚ 1.563615e+06 disk.dm-1.out‚ 139.2890625 disk.drbd1260.out‚ 0.68359375 disk.sda2.out‚0$ disk.nvme0c0n1.in‚19740.80859375* proc.memory.percent‚17.908453941345215 cpu‚25.20517932910492) disk./.usagePercent‚78.76192012040629 disk.sda3.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚117126.04248046875" disk.nvme0n1.in‚19740.80859375# proc.memory.rssMB‚ 2934.12109375 proc.cpu.threads‚17% disk./.usageGB‚1173.6095275878906 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 139.2890625 disk.sda1.in‚0$ memory_percent‚96.56977653503418 disk.dm-0.in‚19740.80859375$ disk.dm-0.out‚117126.04248046875 disk.sda3.out‚ 139.2890625 disk.drbd1260.in‚0Ž¡@’”‚(:‡ ³öèÔ¨û›¯ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0604 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i$ö•‚(:ö ³öèÔ²ÿë³ disk.sda3.in‚0 disk.dm-1.out‚ 139.671875 disk.sda2.out‚0 cpu‚25.204822772358995 proc.cpu.threads‚17 network.sent‚ 1.02342e+07$ disk.nvme0c0n1.in‚19740.80859375 disk.dm-0.in‚19740.80859375$ disk.dm-0.out‚117439.76513671875 disk.sda3.out‚ 139.671875 disk.sda1.in‚0& proc.memory.availableMB‚ 562.015625# proc.memory.rssMB‚ 2934.12109375) disk./.usagePercent‚78.74909030282782) disk.nvme0c0n1.out‚117439.76513671875" disk.nvme0n1.in‚19740.80859375' disk.nvme0n1.out‚117439.76513671875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.68359375 disk.sda.in‚0* proc.memory.percent‚17.908453941345215 network.recv‚ 1.567148e+06 disk.dm-1.in‚0 disk.sda.out‚ 139.671875 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚96.56972885131836% disk./.usageGB‚1173.4183540344238PbM2–- _wandbruntime‚4699  _runtime‚4699=d¸ÔÇ—Á & lr-AdamW/pg2‚0.0006357757952079284 trainer/global_step‚4399 % _timestamp‚1.7884925998318684e+09 _runtime‚4699.472276426 _step‚357 & lr-AdamW/pg1‚0.0006357757952079284ˆ¼BÝÌ˜Æ & lr-AdamW/pg1‚0.0006357757952079284 & lr-AdamW/pg2‚0.0006357757952079284 trainer/global_step‚4399 % _timestamp‚1.7884925998318684e+09 _runtime‚4699.472276426 _step‚357åµÎÀ:2™- _wandbruntime‚4699  _runtime‚4699Ðáò'šìO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76706380 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚311004869212672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76706378 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚298704756366576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚298704773405936 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24412927 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚306494306556416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43965835 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311004078564864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚4699.473841085 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97389154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97412170 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚306494324559360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚306494306556416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97412170 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43965837 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345135689936384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚303199077542184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4509772008448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚311004869212672 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174095532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚303198289854412 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211790848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚311004078564864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4705603573248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345135478145536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4510544653312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76706380 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97389154 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4494304136248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 % _timestamp‚1.7884925998332055e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚358 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24412401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4509772008448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187850752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43965835 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311004078564864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚4699.473841085 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97389154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97412170 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚306494324559360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚306494306556416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97412170 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43965837 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345135689936384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚303199077542184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4509772008448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚311004869212672 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174095532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚303198289854412 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211790848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚311004078564864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4705603573248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345135478145536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4510544653312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76706380 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97389154 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4494304136248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 % _timestamp‚1.7884925998332055e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚358 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24412401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4509772008448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187850752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174157161 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4706652066304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174157161 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76723252 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97433907 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚303266145118030 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚311073573911040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43975485 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4706453743104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884926007059348e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174134117 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24417741 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68393224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97410865 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340505410765824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97410865 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4511549412352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚306562006495744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298770822802336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚359 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340505434705920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992Óì¦'žûM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4511549412352 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚306562024498688 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4510776754176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174157161 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4706652066304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174157161 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76723252 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97433907 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚303266145118030 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚311073573911040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43975485 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4706453743104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884926007059348e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174134117 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24417741 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68393224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97410865 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340505410765824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97410865 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4511549412352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚306562006495744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298770822802336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚359 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340505434705920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚4700.346886012ç[J’S2Ÿ- _wandbruntime‚4700  _runtime‚4700®9^Ÿ^ Ø /loss/FRt1/detection_loss‚2.5408323325894098 0loss/FRt1/assignment_loss‚3.5440633703361857 /loss/FRt3/assignment_loss‚2.003463715314865 0loss/FRt4/assignment_loss‚1.0352517825861771 1loss/FBt2/assignment_loss‚0.13862023932888407 /loss/FBt4/detection_loss‚0.6789977413910332 2loss/SRqqt1/assignment_loss‚1.4734082811100537 /loss/FRt4/detection_loss‚1.6079612279621263 1loss/SRqqt2/detection_loss‚2.1917208088071725 1loss/SRqqt3/detection_loss‚1.3277113012185222 .loss/FBt1/detection_loss‚2.427486591041088 1loss/FBt1/assignment_loss‚0.11646636272742228 .loss/FBt2/detection_loss‚3.195163309574127 0loss/FRt2/assignment_loss‚2.4591910662474454 2loss/SRqqt3/assignment_loss‚1.6947014991586145 trainer/global_step‚4399 % _timestamp‚1.7884926007083893e+09 epoch‚3 .loss/FRt3/detection_loss‚2.207772383022876 1loss/SRqqt1/detection_loss‚2.0213122922320697 1loss/SRqqt4/detection_loss‚1.1492147323376727 /loss/FBt3/detection_loss‚2.6559576229615645 _runtime‚4700.348554727 $loss/total_loss‚40.7798662169602 2loss/FBt3/assignment_loss‚0.017626417306018993 1loss/FBt4/assignment_loss‚0.18512634857813226 .loss/FRt2/detection_loss‚3.289393416157475 2loss/SRqqt2/assignment_loss‚1.8526283733938869 2loss/SRqqt4/assignment_loss‚0.9657950015773751 _step‚360»Ïc¡Ý 0loss/FRt2/assignment_loss‚2.4591910662474454 2loss/SRqqt3/assignment_loss‚1.6947014991586145 trainer/global_step‚4399 % _timestamp‚1.7884926007083893e+09 epoch‚3 .loss/FRt3/detection_loss‚2.207772383022876 1loss/SRqqt1/detection_loss‚2.0213122922320697 1loss/SRqqt4/detection_loss‚1.1492147323376727 /loss/FBt3/detection_loss‚2.6559576229615645 _runtime‚4700.348554727 $loss/total_loss‚40.7798662169602 2loss/FBt3/assignment_loss‚0.017626417306018993 1loss/FBt4/assignment_loss‚0.18512634857813226 .loss/FRt2/detection_loss‚3.289393416157475 2loss/SRqqt2/assignment_loss‚1.8526283733938869 2loss/SRqqt4/assignment_loss‚0.9657950015773751 _step‚360 /loss/FRt1/detection_loss‚2.5408323325894098 0loss/FRt1/assignment_loss‚3.5440633703361857 /loss/FRt3/assignment_loss‚2.003463715314865 0loss/FRt4/assignment_loss‚1.0352517825861771 1loss/FBt2/assignment_loss‚0.13862023932888407 /loss/FBt4/detection_loss‚0.6789977413910332 2loss/SRqqt1/assignment_loss‚1.4734082811100537 /loss/FRt4/detection_loss‚1.6079612279621263 1loss/SRqqt2/detection_loss‚2.1917208088071725 1loss/SRqqt3/detection_loss‚1.3277113012185222 .loss/FBt1/detection_loss‚2.427486591041088 1loss/FBt1/assignment_loss‚0.11646636272742228 .loss/FBt2/detection_loss‚3.195163309574127èA‹h@’¢‚(:‡ ÂöèÔׇî¬ gpu.0.gpu‚18.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9564 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬¼ ƒ£‚(:ø ÂöèÔÈ’˜± disk.sda1.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 561.97265625 cpu‚25.204418755758976 network.recv‚ 1.570587e+06) disk./.usagePercent‚78.79980718741358 disk.sda3.in‚0 disk.dm-1.out‚ 140.0703125 disk.drbd1260.out‚0.6875 disk.sda1.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚19740.80859375 network.sent‚ 1.0239173e+07 disk.dm-1.in‚0) disk.nvme0c0n1.out‚117896.50048828125" disk.nvme0n1.in‚19740.80859375 disk.dm-0.in‚19740.80859375* proc.memory.percent‚17.908453941345215 proc.cpu.threads‚17$ disk./.usageGB‚1174.174072265625 disk.sda3.out‚ 140.0703125 disk.drbd1260.in‚0 disk.sda.in‚0' disk.nvme0n1.out‚117896.50048828125$ disk.dm-0.out‚117896.50048828125$ memory_percent‚96.56999111175537# proc.memory.rssMB‚ 2934.12109375 disk.sda.out‚ 140.0703125‹ãÕ¤‚(:’ ÑöèÔ˜ó€° gpu.0.gpu‚41.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.897' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.379399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Zײ„¥‚(:ù ÑöèÔ²Ðâ°$ memory_percent‚96.56846523284912( proc.memory.availableMB‚ 562.22265625# proc.memory.rssMB‚ 2934.12109375% disk./.usageGB‚1174.1664810180664# disk.nvme0c0n1.in‚ 19858.2890625! disk.nvme0n1.in‚ 19858.2890625 disk.dm-1.out‚ 140.54296875 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 140.54296875 disk.sda2.out‚0) disk./.usagePercent‚78.79929773242044) disk.nvme0c0n1.out‚118214.26611328125$ disk.dm-0.out‚118214.26611328125 disk.drbd1260.out‚0.6875 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.908453941345215 cpu‚25.204097579755388 network.recv‚ 1.577641e+06' disk.nvme0n1.out‚118214.26611328125 disk.sda3.in‚0 proc.cpu.threads‚17 network.sent‚ 1.0350307e+07 disk.dm-0.in‚ 19858.2890625 disk.sda3.out‚ 140.54296875 disk.dm-1.in‚0 disk.sda2.in‚0uÞ½ø¦‚(:… àöèÔŠç—¯ gpu.0.gpu‚40.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÝGŸ}§‚(:ò àöèÔ—çá± network.sent‚ 1.0356181e+07) disk./.usagePercent‚78.80083710572382# disk.nvme0c0n1.in‚ 19858.2890625# disk.dm-0.out‚118713.1103515625 disk.sda3.out‚ 141.0234375 disk.dm-1.in‚0' proc.memory.availableMB‚ 562.2109375# proc.memory.rssMB‚ 2934.12109375% disk./.usageGB‚1174.1894187927246 disk.sda2.out‚0! disk.nvme0n1.in‚ 19858.2890625 disk.sda1.out‚0$ memory_percent‚96.56853675842285* proc.memory.percent‚17.908453941345215 disk.sda2.in‚0& disk.nvme0n1.out‚118713.1103515625 disk.drbd1260.out‚ 0.69140625 disk.sda.in‚0 cpu‚25.203814374934 proc.cpu.threads‚17 network.recv‚ 1.58196e+06( disk.nvme0c0n1.out‚118713.1103515625 disk.dm-0.in‚ 19858.2890625 disk.sda3.in‚0 disk.dm-1.out‚ 141.0234375 disk.drbd1260.in‚0 disk.sda.out‚ 141.0234375 disk.sda1.in‚0‰}¼Ì2¨- _wandbruntime‚4749  _runtime‚4749‚f‹˜Ç©Á % _timestamp‚1.7884926498412828e+09 _runtime‚4749.481581423 _step‚361 & lr-AdamW/pg1‚0.0006311978154897237 & lr-AdamW/pg2‚0.0006311978154897237 trainer/global_step‚4449¦)[ÌªÆ & lr-AdamW/pg1‚0.0006311978154897237 & lr-AdamW/pg2‚0.0006311978154897237 trainer/global_step‚4449 % _timestamp‚1.7884926498412828e+09 _runtime‚4749.481581423 _step‚361éý %†2«- _wandbruntime‚4749  _runtime‚4749Ád»~¯¬ìO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4560796353536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98478056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348955739871744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 trainer/global_step‚4449 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 _runtime‚4749.482779708 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77550080 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4560023708672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚314439327231488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚306551682227520 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24677959 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4758236369408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4560796353536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚309879303522816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24678506 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 % _timestamp‚1.7884926498422823e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4543604072944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77550080 _step‚362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚309879321525760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69126743 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77550078 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77550078 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348955502915072 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44448235 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314440117879296 GADeviceStatsMonitor.on_train_batch_staÌqgCrt/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302008078154576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4560023708672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4544374721356 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176028134 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98478056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44448237 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚309879303522816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176051152 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚314440117879296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98501072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 213016576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69126194 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314439327231488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚344197479562240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚306552469915292 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4758023352832 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302008095193936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 J¢÷'­ñO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24677959 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4758236369408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4560796353536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚309879303522816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24678506 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 % _timestamp‚1.7884926498422823e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4543604072944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77550080 _step‚362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚309879321525760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69126743 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77550078 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77550078 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348955502915072 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44448235 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314440117879296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302008078154576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4560023708672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4544374721356 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176028134 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98478056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44448237 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚309879303522816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176051152 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚314440117879296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98501072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 213016576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69126194 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314439327231488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚344197479562240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚306552469915292 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4758023352832 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302008095193936 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4560796353536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98478056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348955739871744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 trainer/global_step‚4449 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 _runtime‚4749.482779708 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77550080 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4560023708672 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚314439327231488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚306551682227520 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825êît/^2®- _wandbruntime‚4750  _runtime‚4750zÓÝàü&¯öM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4561801112576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69141753 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69141211 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚363 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98499767 _runtime‚4750.397278901 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚314508822577664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚314508822577664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚309947021465088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77566952 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚309947003462144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4759079511552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚306618749803158 % _timestamp‚1.7884926507563605e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176066719 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77566952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4545375861442 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4544605212822 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚314508031916544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98499767 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176089763 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚349032125485568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚306619537491138 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚309947003462144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98522809 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302074144590336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4561028454400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4759273640448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4561028454400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4561801112576 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302074161629696 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚344272827905024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77566954 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0t”š¹u°ûM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4561801112576 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302074161629696 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚344272827905024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77566954 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4561801112576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69141753 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69141211 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚363 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98499767 _runtime‚4750.397278901 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚314508822577664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚314508822577664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚309947021465088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77566952 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚309947003462144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4759079511552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚306618749803158 % _timestamp‚1.7884926507563605e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176066719 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77566952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4545375861442 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4544605212822 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚314508031916544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_traiPÏùvŒ n_batch_end/allocation.small_pool.freed‚98499767 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176089763 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚349032125485568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚306619537491138 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚309947003462144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98522809 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302074144590336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4561028454400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4759273640448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4561028454400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620ëv}2±- _wandbruntime‚4750  _runtime‚4750¯.g_²Ù 0loss/FRt1/assignment_loss‚3.0836068188602272 2loss/FBt2/assignment_loss‚0.031092369014838066 1loss/FBt3/assignment_loss‚0.46171022870839806 /loss/FRt4/detection_loss‚1.7886544091254473 0loss/SRqqt3/detection_loss‚1.306225172731848 1loss/FBt4/assignment_loss‚0.12117471666874327 2loss/SRqqt1/assignment_loss‚1.7138011420611292 2loss/SRqqt4/assignment_loss‚0.7932445028323739 /loss/FBt1/detection_loss‚1.7821282545725505 /loss/FBt3/detection_loss‚2.3544597923755646 2loss/SRqqt2/assignment_loss‚1.6056518407538534 /loss/FBt4/detection_loss‚0.6887035532544057 _step‚364 /loss/FRt2/assignment_loss‚3.772974563272376 % _timestamp‚1.7884926507586246e+09 /loss/FRt4/assignment_loss‚1.111877633333206 1loss/SRqqt1/detection_loss‚2.4771732464432716 0loss/SRqqt2/detection_loss‚2.254778442531824 1loss/SRqqt4/detection_loss‚0.9727808860914101 _runtime‚4750.398791988 /loss/FRt3/assignment_loss‚2.272515184000919 1loss/FBt1/assignment_loss‚0.33120071280670044 .loss/FRt1/detection_loss‚2.483981844376434 /loss/FRt3/detection_loss‚1.9241038390941787 &loss/total_loss‚40.158548106485526 2loss/SRqqt3/assignment_loss‚1.3225675002294814 /loss/FRt2/detection_loss‚2.2138383820064162 /loss/FBt2/detection_loss‚3.2903030713399253 epoch‚3 trainer/global_step‚4449’Þ<ïd³Þ trainer/global_step‚4449 0loss/FRt1/assignment_loss‚3.0836068188602272 2loss/FBt2/assignment_loss‚0.031092369014838066 1loss/FBt3/assignment_loss‚0.46171022870839806 /loss/FRt4/detection_loss‚1.7886544091254473 0loss/SRqqt3/detection_loss‚1.306225172731848 1loss/FBt4/assignment_loss‚0.12117471666874327 2loss/SRqqt1/assignment_loss‚1.7138011420611292 2loss/SRqqt4/assignment_loss‚0.7932445028323739 /loss/FBt1/detection_loss‚1.7821282545725505 /loss/FBt3/detection_loss‚2.3544597923755646 2loss/SRqqt2/assignment_loss‚1.6056518407538534 /loss/FBt4/detection_loss‚0.6887035532544057 _step‚364 /loss/FRt2/assignment_loss‚3.772974563272376 % _timestamp‚1.7884926507586246e+09 /loss/FRt4/assignment_loss‚1.111877633333206 1loss/SRqqt1/detection_loss‚2.4771732464432716 0loss/SRqqt2/detection_loss‚2.254778442531824 1loss/SRqqt4/detection_loss‚0.9727808860914101 _runtime‚4750.398791988 /loss/FRt3/assignment_loss‚2.272515184000919 1loss/FBt1/assignment_loss‚0.33120071280670044 .loss/FRt1/detection_loss‚2.483981844376434 /loss/FRt3/detection_loss‚1.9241038390941787 &loss/total_loss‚40.158548106485526 2loss/SRqqt3/assignment_loss‚1.3225675002294814 /loss/FRt2/detection_loss‚2.2138383820064162 /loss/FBt2/detection_loss‚3.2903030713399253 epoch‚3ìŒ^™R’´‚(:‡ ïöèÔ¦Ÿ˜² gpu.0.gpu‚40.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.6316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é-´0‚µ‚(:÷ ïöèÔÞÝî´ network.sent‚ 1.0359367e+07# disk.dm-0.out‚119006.7783203125 disk.sda3.in‚0 disk.sda3.out‚ 141.5078125 disk.dm-1.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 561.55078125 disk.sda2.in‚0( disk.nvme0c0n1.out‚119006.7783203125! disk.nvme0n1.in‚ 19858.2890625& disk.nvme0n1.out‚119006.7783203125$ memory_percent‚96.57256603240967 network.recv‚ 1.585628e+06) disk./.usagePercent‚78.80083710572382% disk./.usageGB‚1174.1894187927246 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.69140625 disk.sda.out‚ 141.5078125# proc.memory.rssMB‚ 2934.12109375# disk.nvme0c0n1.in‚ 19858.2890625 disk.dm-0.in‚ 19858.2890625 disk.dm-1.out‚ 141.5078125 disk.sda.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.908453941345215 cpu‚25.203459780070677 proc.cpu.threads‚17M¯ûp’¶‚(:‡ þöèÔó°ó¯ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.427' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸†½u·‚(:ê þöèԪ㪳) disk./.usagePercent‚78.80083710572382" disk.dm-0.out‚119156.595703125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.6953125 network.sent‚ 1.046924e+07! disk.nvme0n1.in‚ 19858.2890625 disk.dm-0.in‚ 19858.2890625 disk.sda.out‚ 141.84375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚96.57049179077148# proc.memory.rssMB‚ 2934.12109375% disk.nvme0n1.out‚119156.595703125 disk.dm-1.in‚0 disk.dm-1.out‚ 141.84375 disk.sda2.in‚0& proc.memory.availableMB‚ 561.890625* proc.memory.percent‚17.908453941345215 cpu‚25.203155380493655 proc.cpu.threads‚17% disk./.usageGB‚1174.1894187927246# disk.nvme0c0n1.in‚ 19858.2890625' disk.nvme0c0n1.out‚119156.595703125 disk.sda3.out‚ 141.84375 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.out‚0 network.recv‚ 1.591462e+06tr*m’¸‚(:‡ ÷èÔòýŸ® gpu.0.gpu‚17.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.354' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î/ƒ¹‚(:ø ÷èÔáÒÕ® disk.sda2.out‚0$ memory_percent‚96.57046794891357( proc.memory.availableMB‚ 561.89453125 network.recv‚ 1.596322e+06% disk./.usageGB‚1174.1856384277344# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215$ disk.nvme0c0n1.in‚20006.81640625% disk.nvme0n1.out‚119480.416015625 disk.dm-0.in‚20006.81640625 disk.dm-1.out‚ 142.22265625 disk.sda.in‚0 disk.sda.out‚ 142.22265625 cpu‚25.20278490430388 proc.cpu.threads‚17 network.sent‚ 1.0473586e+07) disk./.usagePercent‚78.80058340225739" disk.nvme0n1.in‚20006.81640625 disk.sda3.in‚0 disk.sda3.out‚ 142.22265625 disk.drbd1260.out‚ 0.6953125 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚119480.416015625" disk.dm-0.out‚119480.416015625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 ½Œ2º- _wandbruntime‚4799  _runtime‚4799µíچƻÀ _step‚365 & lr-AdamW/pg1‚0.0006265708461267923 & lr-AdamW/pg2‚0.0006265708461267923 trainer/global_step‚4499 $ _timestamp‚1.788492700075438e+09 _runtime‚4799.715929124ÕÀÁ2˼Š_step‚365 & lr-AdamW/pg1‚0.0006265708461267923 & lr-AdamW/pg2‚0.0006265708461267923 trainer/global_step‚4499 $ _timestamp‚1.788492700075438e+09 _runtime‚4799.715929124ívÈ×–2½- _wandbruntime‚4799  _runtime‚4799ôŸ>ò'¾ìO _runtime‚4799.717102803 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317874575898112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313264300489216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78393778 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69874019 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚317875366545920 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177960736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78393778 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313264318492160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4810451736576 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352775564423168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305311416981936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313264318492160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4611048053760 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177983754 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309905074600628 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 _step‚366 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309905862288400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44930637 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚317874575898112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78393780 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347964896701440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99589974 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99566958 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352775348438016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4611048053760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99566958 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44930635 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24943384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4594445306464 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4610275408896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305311399942576·“ñG÷'¿ñO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4611048053760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99566958 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44930635 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24943384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4594445306464 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4610275408896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305311399942576 _runtime‚4799.717102803 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317874575898112 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313264300489216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78393778 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69874019 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚317875366545920 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177960736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78393778 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313264318492160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4810451736576 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352775564423168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305311416981936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313264318492160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4611048053760 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177983754 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309905074600628 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 _step‚366 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309905862288400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44930637 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚317874575898112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78393780 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347964896701440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99589974 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99566958 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352775348438016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360î¸@p’À‚(:‡ œ÷èÔêÇí¯ gpu.0.gpu‚33.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øƒˆÁ‚(:ý œ÷èÔ®’±% disk./.usageGB‚1174.1738929748535 disk.drbd1260.in‚0 disk.sda1.out‚0' disk.nvme0n1.out‚119815.76708984375 disk.sda3.out‚ 142.72265625( proc.memory.availableMB‚ 561.64453125* proc.memory.percent‚17.908453941345215 cpu‚25.202493738019 disk.sda2.out‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚119815.01708984375" disk.nvme0n1.in‚20006.81640625 disk.dm-0.in‚20006.81640625$ memory_percent‚96.57199382781982 network.sent‚ 1.0479532e+07 disk.drbd1260.out‚ 0.69921875$ disk.nvme0c0n1.in‚20006.81640625 disk.sda3.in‚0# proc.memory.rssMB‚ 2934.12109375 proc.cpu.threads‚17 network.recv‚ 1.600374e+06) disk./.usagePercent‚78.79979515505946 disk.dm-1.in‚0 disk.dm-1.out‚ 142.72265625 disk.sda.in‚0 disk.sda.out‚ 142.72265625 disk.sda2.in‚0$ disk.dm-0.out‚119791.51708984375îFËm2Â- _wandbruntime‚4800  _runtime‚4800jb+™ÃöM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178022365 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚317943280583168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚317944071244288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305377466378336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177999321 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313332018431488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4612052812800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313332018431488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99588669 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78410654 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4811499219968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313332000428544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178022365 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177999321 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352851964430336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305377483417696 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚367 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69889001 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78410652 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚317944071244288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 trainer/global_step‚4499 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚309972929864246 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24948718 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348040245044224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78410654 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352851744264192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99588669 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4611280154624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99611711 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313332000428544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309972142176266 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005&ÈóS'ÄûM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚309972929864246 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24948718 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348040245044224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78410654 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352851744264192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99588669 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4611280154624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99611711 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313332000428544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309972142176266 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178022365 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚317943280583168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚317944071244288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305377466378336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177999321 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313332018431488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4612052812800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313332018431488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99588669 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78410654 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4811499219968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313332000428544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178022365 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177999321 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352851964430336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305377483417696 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚367 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69889001 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78410652 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚317944071244288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 trainer/global_step‚4499 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944w2Å- _wandbruntime‚4800  _runtime‚4800Ÿý!MWÆÑ /loss/FRt1/assignment_loss‚3.140643166773247 1loss/FBt3/assignment_loss‚0.09903653351752187 1loss/FBt4/assignment_loss‚0.11859513859717079 /loss/FRt4/detection_loss‚1.6236821146061022 /loss/FBt4/detection_loss‚0.5959540384604587 %loss/total_loss‚42.26059980155012 /loss/FRt2/assignment_loss‚3.047841747105122 0loss/FRt3/assignment_loss‚2.3229862506999526 1loss/FBt1/assignment_loss‚0.06853591342657567 1loss/SRqqt4/detection_loss‚1.4671778111563374 .loss/FBt3/detection_loss‚3.210376364844186 % _timestamp‚1.7884927009458654e+09 /loss/SRqqt1/assignment_loss‚1.1925388907685 2loss/SRqqt2/assignment_loss‚1.6344936644589458 2loss/SRqqt3/assignment_loss‚1.2494664215418982 /loss/FBt2/assignment_loss‚0.172386990612722 _runtime‚4800.586071688 trainer/global_step‚4499 .loss/FRt3/detection_loss‚2.312023267496464 1loss/SRqqt1/detection_loss‚2.4263003082836376 /loss/FBt1/detection_loss‚2.6431275606155396 0loss/FRt4/assignment_loss‚1.8496260605752468 /loss/FRt2/detection_loss‚2.3866882953204116 /loss/SRqqt3/detection_loss‚1.50499645008208 .loss/FBt2/detection_loss‚2.773341193795204 2loss/SRqqt4/assignment_loss‚1.1283907830559958 /loss/FRt1/detection_loss‚2.6446736071145898 1loss/SRqqt2/detection_loss‚2.6477172286422164 epoch‚3 _step‚368®J¯Ê\ÇÖ trainer/global_step‚4499 .loss/FRt3/detection_loss‚2.312023267496464 1loss/SRqqt1/detection_loss‚2.4263003082836376 /loss/FBt1/detection_loss‚2.6431275606155396 0loss/FRt4/assignment_loss‚1.8496260605752468 /loss/FRt2/detection_loss‚2.3866882953204116 /loss/SRqqt3/detection_loss‚1.50499645008208 .loss/FBt2/detection_loss‚2.773341193795204 2loss/SRqqt4/assignment_loss‚1.1283907830559958 /loss/FRt1/detection_loss‚2.6446736071145898 1loss/SRqqt2/detection_loss‚2.6477172286422164 epoch‚3 _step‚368 /loss/FRt1/assignment_loss‚3.140643166773247 1loss/FBt3/assignment_loss‚0.09903653351752187 1loss/FBt4/assignment_loss‚0.11859513859717079 /loss/FRt4/detection_loss‚1.6236821146061022 /loss/FBt4/detection_loss‚0.5959540384604587 %loss/total_loss‚42.26059980155012 /loss/FRt2/assignment_loss‚3.047841747105122 0loss/FRt3/assignment_loss‚2.3229862506999526 1loss/FBt1/assignment_loss‚0.06853591342657567 1loss/SRqqt4/detection_loss‚1.4671778111563374 .loss/FBt3/detection_loss‚3.210376364844186 % _timestamp‚1.7884927009458654e+09 /loss/SRqqt1/assignment_loss‚1.1925388907685 2loss/SRqqt2/assignment_loss‚1.6344936644589458 2loss/SRqqt3/assignment_loss‚1.2494664215418982 /loss/FBt2/assignment_loss‚0.172386990612722 _runtime‚4800.586071688ð«²+3È‚(:’ «÷èÔÝ—Ô­ gpu.0.gpu‚37.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.454' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.890800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J‡ z‡É‚(:ü «÷èԮʂ° disk.drbd1260.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚120108.11083984375' proc.memory.availableMB‚ 561.3515625( disk./.usagePercent‚78.7697073575828" disk.nvme0n1.in‚20006.81640625 disk.sda.in‚0$ disk.nvme0c0n1.in‚20006.81640625# proc.memory.rssMB‚ 2934.12109375 cpu‚25.202242756682203 proc.cpu.threads‚17 disk.dm-0.in‚20006.81640625 disk.sda3.in‚0 disk.drbd1260.out‚ 0.69921875 disk.sda.out‚ 143.05859375 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.908453941345215 network.sent‚ 1.057615e+07 network.recv‚ 1.60568e+06% disk./.usageGB‚1173.7255630493164' disk.nvme0n1.out‚120108.11083984375 disk.dm-1.out‚ 143.05859375 disk.sda1.out‚0$ memory_percent‚96.57378196716309$ disk.dm-0.out‚120108.11083984375 disk.sda3.out‚ 143.05859375 disk.dm-1.in‚0Hhªó’Ê‚(:‡ º÷èÔÈ”‚¯ gpu.0.gpu‚36.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.364' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.2728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k­†Ë‚(:û º÷èÔúâײ( disk.nvme0c0n1.out‚120524.6572265625" disk.nvme0n1.in‚20006.81640625# proc.memory.rssMB‚ 2934.12109375 disk.sda3.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 561.6015625* proc.memory.percent‚17.908453941345215 cpu‚25.20190218587316 proc.cpu.threads‚17 network.recv‚ 1.611016e+06) disk./.usagePercent‚78.77652048609154 disk.sda1.out‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚20006.81640625& disk.nvme0n1.out‚120524.6572265625 disk.dm-0.in‚20006.81640625# disk.dm-0.out‚120524.6572265625% disk./.usageGB‚1173.8270835876465 disk.sda3.out‚ 143.58984375 disk.dm-1.in‚0 disk.dm-1.out‚ 143.58984375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.69921875 disk.sda.in‚0$ memory_percent‚96.57225608825684 network.sent‚ 1.0673388e+07 disk.sda.out‚ 143.58984375 disk.sda1.in‚0Ë\%¯Ì‚(:’ É÷èÔòŒÐ½ gpu.0.gpu‚33.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.816' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.363200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ^6öÍ‚(:ô É÷èÔó´£Î disk.drbd1260.in‚0 disk.drbd1260.out‚0.703125# disk.dm-0.out‚120896.4228515625 disk.dm-1.in‚0 disk.dm-1.out‚ 143.92578125$ memory_percent‚96.56920433044434' proc.memory.availableMB‚ 562.1015625* proc.memory.percent‚17.908453941345215" disk.nvme0c0n1.in‚ 20113.859375( disk.nvme0c0n1.out‚120896.4228515625 network.recv‚ 1.615924e+06) disk./.usagePercent‚78.78651886034129% disk./.usageGB‚1173.9760665893555 disk.sda1.out‚0 disk.sda3.out‚ 143.92578125 cpu‚25.201606043358446 network.sent‚ 1.0679671e+07 disk.sda2.out‚0& disk.nvme0n1.out‚120896.4228515625 disk.dm-0.in‚ 20113.859375 disk.sda3.in‚0# proc.memory.rssMB‚ 2934.12109375 disk.sda.in‚0 disk.sda.out‚ 143.92578125 disk.sda1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 20113.859375 proc.cpu.threads‚17î‹l2Î- _wandbruntime‚4850  _runtime‚4850|ÀÓÇÏÁ trainer/global_step‚4549 % _timestamp‚1.7884927510903096e+09 _runtime‚4850.730916787 _step‚369 & lr-AdamW/pg1‚0.0006218958513726145 & lr-AdamW/pg2‚0.0006218958513726145ÿê_ÌÐÆ & lr-AdamW/pg1‚0.0006218958513726145 & lr-AdamW/pg2‚0.0006218958513726145 trainer/global_step‚4549 % _timestamp‚1.7884927510903096e+09 _runtime‚4850.730916787 _step‚369ñ‡ž¹O2Ñ- _wandbruntime‚4850  _runtime‚4850Pô¸vøÒïO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100655860 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4862838002688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚313258466973736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70622691 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70623222 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚356595151843328 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚313259254661508 _runtime‚4850.731982085 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79237480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100678876 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316649297455616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4661299753984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179916356 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100678876 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4863032144896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4644515891572 trainer/global_step‚4549 _step‚370 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321310615212544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4643745243160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321309824564736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100655860 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 $ _timestamp‚1.788492751091452e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308614738769936 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4660527109120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚321310615212544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79237478 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179893338 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4660527109120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45413035 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356595369925632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 aMDeviceStatsM7XFý onitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚351732313840640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79237478 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316649315458560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4661299753984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316649315458560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45413037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321309824564736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316649297455616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25209656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351732337780736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321309824564736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316649297455616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25209656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351732337780736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70622691 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70623222 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚356595151843328 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚313259254661508 _runtime‚4850.731982085 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79237480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100678876 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316649297455616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4661299753984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179916356 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100678876 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4863032144896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4644515891572 trainer/global_step‚4549 _step‚370 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321310615212544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4643745243160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321309824564736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100655860 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 $ _timestamp‚1.788492751091452e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308614738769936 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4660527109120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚321310615212544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79237478 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179893338 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4660527109120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45413035 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356595369925632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚351732313840640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79237478 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316649315458560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 373079040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4661299753984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316649315458560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45413037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0òË r‚2Ô- _wandbruntime‚4851  _runtime‚4851Tð<þ&ÕøM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚313326322237354 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316716997394944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45422685 _step‚371 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316717015397888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45422683 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79254352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4661531854848 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79254354 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚313325534549374 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 _runtime‚ 4851.67538914 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316717015397888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4662304513024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 trainer/global_step‚4549 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316716997394944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4864081343488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100700613 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100677571 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4645517031658 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79254354 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308680805205696 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179931923 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179954967 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4661531854848 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 $ _timestamp‚1.788492752034087e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308680788166336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179931923 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321379319910912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25214908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4662304513024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70637591 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4863885117440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚356671547300864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351807686123520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79254352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351807662183424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100700613 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100677571H©«G¶"ÖýM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100677571 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4645517031658 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79254354 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 373079040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308680805205696 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179931923 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179954967 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4661531854848 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 $ _timestamp‚1.788492752034087e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308680788166336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179931923 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321379319910912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25214908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4662304513024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70637591 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4863885117440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚356671547300864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351807686123520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79254352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351807662183424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100700613 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100677571 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚313326322237354 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316716997394944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45422685 _step‚371 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316717015397888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45422683 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79254352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4661531854848 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79254354 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚313325534549374 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 _runtime‚ 4851.67538914 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316717015397888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.alloca"KÿMted‚ 4662304513024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 trainer/global_step‚4549 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316716997394944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4864081343488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100700613ó¿{ëÈ2×- _wandbruntime‚4851  _runtime‚4851Û¦9^ØØ /loss/FRt1/detection_loss‚2.4371889477243296 0loss/SRqqt1/detection_loss‚2.092122292518616 /loss/FBt2/detection_loss‚2.9484459246907915 _step‚372 0loss/FRt4/assignment_loss‚1.3347228439349048 1loss/SRqqt2/assignment_loss‚1.422212153673172 /loss/FRt2/detection_loss‚2.3074101017381885 1loss/SRqqt2/detection_loss‚2.8962284786005816 1loss/SRqqt3/detection_loss‚1.6030411201739503 _runtime‚4851.677711998 1loss/SRqqt4/detection_loss‚0.8649974249536172 .loss/FBt1/detection_loss‚2.798588164150715 .loss/FBt3/detection_loss‚2.237311111556159 %loss/total_loss‚42.26213102540908 1loss/FBt1/assignment_loss‚0.06681736069729549 /loss/FRt4/detection_loss‚1.6512581965060167 % _timestamp‚1.7884927520374508e+09 2loss/SRqqt3/assignment_loss‚1.7543473860489265 1loss/FBt4/assignment_loss‚0.14023054545825223 /loss/FRt3/detection_loss‚2.5285760742567835 trainer/global_step‚4549 /loss/FRt3/assignment_loss‚2.702439296813238 2loss/SRqqt1/assignment_loss‚1.4436058273272856 2loss/SRqqt4/assignment_loss‚1.0645816882140935 /loss/FBt4/detection_loss‚0.4244495351905809 0loss/FRt1/assignment_loss‚3.0695680283211373 1loss/FBt3/assignment_loss‚0.29341914319381757 epoch‚3 /loss/FRt2/assignment_loss‚4.161993530488783 2loss/FBt2/assignment_loss‚0.018575849177848016SÌ5cÙÝ .loss/FBt1/detection_loss‚2.798588164150715 .loss/FBt3/detection_loss‚2.237311111556159 %loss/total_loss‚42.26213102540908 1loss/FBt1/assignment_loss‚0.06681736069729549 /loss/FRt4/detection_loss‚1.6512581965060167 % _timestamp‚1.7884927520374508e+09 2loss/SRqqt3/assignment_loss‚1.7543473860489265 1loss/FBt4/assignment_loss‚0.14023054545825223 /loss/FRt3/detection_loss‚2.5285760742567835 trainer/global_step‚4549 /loss/FRt3/assignment_loss‚2.702439296813238 2loss/SRqqt1/assignment_loss‚1.4436058273272856 2loss/SRqqt4/assignment_loss‚1.0645816882140935 /loss/FBt4/detection_loss‚0.4244495351905809 0loss/FRt1/assignment_loss‚3.0695680283211373 1loss/FBt3/assignment_loss‚0.29341914319381757 epoch‚3 /loss/FRt2/assignment_loss‚4.161993530488783 2loss/FBt2/assignment_loss‚0.018575849177848016 /loss/FRt1/detection_loss‚2.4371889477243296 0loss/SRqqt1/detection_loss‚2.092122292518616 /loss/FBt2/detection_loss‚2.9484459246907915 _step‚372 0loss/FRt4/assignment_loss‚1.3347228439349048 1loss/SRqqt2/assignment_loss‚1.422212153673172 /loss/FRt2/detection_loss‚2.3074101017381885 1loss/SRqqt2/detection_loss‚2.8962284786005816 1loss/SRqqt3/detection_loss‚1.6030411201739503 _runtime‚4851.677711998 1loss/SRqqt4/detection_loss‚0.8649974249536172ô¤Xnò’Ú‚(:‡ Ø÷èÔõŽÔ® gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚173.736' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7472 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±Gˆ$Û‚(:ö Ø÷èÔôüí¯( disk.nvme0c0n1.out‚121191.3916015625( proc.memory.availableMB‚ 561.67578125 network.recv‚ 1.620078e+06# disk.dm-0.out‚121191.3916015625 disk.sda3.in‚0 disk.sda3.out‚ 144.48046875 disk.sda2.out‚0$ memory_percent‚96.57180309295654 cpu‚25.201363551611358 network.sent‚ 1.0685572e+07( disk./.usagePercent‚78.7849241894105 disk.dm-1.out‚ 144.48046875 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 20154.0390625# proc.memory.rssMB‚ 2934.12109375 proc.cpu.threads‚17$ disk./.usageGB‚1173.952304840088 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.908453941345215! disk.nvme0n1.in‚ 20154.0390625& disk.nvme0n1.out‚121191.3916015625 disk.dm-0.in‚ 20154.0390625 disk.drbd1260.out‚0.703125 disk.sda.in‚0 disk.sda.out‚ 144.48046875ëel\Ü‚(:’ ç÷èÔˆ¥õ® gpu.0.gpu‚22.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.536' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.907200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sQñŸ}Ý‚(:ò ç÷èÔ³–Ó° disk.sda.out‚ 144.765625 disk.sda1.out‚0( disk.nvme0c0n1.out‚121595.1416015625# disk.dm-0.out‚121595.1416015625 network.sent‚ 1.0793708e+07) disk./.usagePercent‚78.69056211635736 disk.sda3.out‚ 144.765625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 20154.0390625& disk.nvme0n1.out‚121595.1416015625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.7109375 disk.sda2.in‚0! disk.nvme0n1.in‚ 20154.0390625# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 proc.cpu.threads‚17 network.recv‚ 1.625979e+06 disk.sda3.in‚0 disk.dm-1.out‚ 144.765625 disk.sda1.in‚0 disk.dm-0.in‚ 20154.0390625$ memory_percent‚96.57347202301025( proc.memory.availableMB‚ 561.40234375 cpu‚25.201048557815135$ disk./.usageGB‚1172.546241760254˜dñ¡’Þ‚(:‡ ö÷èԧ堮 gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚130.167' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸s}}ß‚(:ò ö÷èÔ¸Œž¯# disk.nvme0c0n1.in‚ 20154.0390625 disk.sda.in‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 20154.0390625# disk.dm-0.out‚122164.7705078125$ memory_percent‚96.57297134399414& proc.memory.availableMB‚ 561.484375# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7109375 disk.sda2.in‚0( disk.nvme0c0n1.out‚122164.7705078125& disk.nvme0n1.out‚122164.7705078125 disk.dm-0.in‚ 20154.0390625 network.sent‚ 1.0800713e+07( disk./.usagePercent‚78.6889467088163 disk.dm-1.out‚ 145.1953125 disk.sda2.out‚0 cpu‚25.20070821011057 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚ 1.629447e+06% disk./.usageGB‚1172.5221710205078 disk.sda3.in‚0 disk.sda3.out‚ 145.1953125 disk.dm-1.in‚0 disk.sda.out‚ 145.1953125Ïîº2à- _wandbruntime‚4898  _runtime‚4898ùá5qÇáÁ _runtime‚4898.677349292 _step‚373 & lr-AdamW/pg1‚0.0006171738054890896 & lr-AdamW/pg2‚0.0006171738054890896 trainer/global_step‚4599 % _timestamp‚1.7884927990369003e+09¥3 `ÌâÆ & lr-AdamW/pg2‚0.0006171738054890896 trainer/global_step‚4599 % _timestamp‚1.7884927990369003e+09 _runtime‚4898.677349292 _step‚373 & lr-AdamW/pg1‚0.0006171738054890896õg¸wð2ã- _wandbruntime‚4898  _runtime‚4898½¶ö'äðO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 % _timestamp‚1.7884927990381882e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4711551454208 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚360415038590464 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101767778 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101744762 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚360415258769920 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320034312424960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4694586476680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80081178 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80081178 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25474437 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324745073231360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181848958 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4693815828268 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4710778809344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71369872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4915503849984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚311918060557936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚316612647034616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4710778809344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45895437 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 trainer/global_step‚4599 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80081180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚324745863879168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚320034294422016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25474966 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80081180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71370403 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45895435 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320034294422016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181825940 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚320034312424960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101744762 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4915307610624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992ä^Èéû'åõO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320034294422016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181825940 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚320034312424960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101744762 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4915307610624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 % _timestamp‚1.7884927990381882e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4711551454208 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚360415038590464 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101767778 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101744762 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚360415258769920 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320034312424960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4694586476680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80081178 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80081178 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25474437 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324745073231360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181848958 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4693815828268 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4710778809344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71369872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4915503849984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚311918060557936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚316612647034616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4710778809344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45895437 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 trainer/global_step‚4599 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80081180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚324745863879168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚320034294422016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25474966 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80081180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71370403 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45895435ö+*¼=2æ- _wandbruntime‚4899  _runtime‚4899õJÜ‘ìçúM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚311984126993696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4694816968146 _runtime‚4899.633173654 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181864525 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80098052 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚316679714610462 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 % _timestamp‚1.7884927999919012e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚360491658329600 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80098054 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25480261 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚355575103262720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4599 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4712556213248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101789515 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71385346 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H@DeviceStaïH'étsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320102012364288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181864525 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚320102012364288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80098054 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚360491433969152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚320101994361344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4916555066880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4711783555072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320101994361344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚324814568577536 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181887569 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101789515 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚324813777916416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚355575079322624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101766473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚311984109954336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚375 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101766473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25479725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚324814568577536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45905083 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4695587616766 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71384808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4916354646528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚316678926922482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80098052 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181887569/oâ¦'èÿM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚360491658329600 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80098054 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25480261 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚355575103262720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4599 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4712556213248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101789515 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71385346 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320102012364288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181864525 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚320102012364288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80098054 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚360491433969152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚320101994361344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4916555066880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4711783555072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320101994361344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚324814568577536 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181887569 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101789515 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚324813777916416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚355575079322624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101766473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚311984109954336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚375 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101766473 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25479725 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚324814568577536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45905083 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4695587616766 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71384808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4916354646528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚316678926922482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80098052 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181887569 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚311984126993696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4694816968146 _runtime‚4899.633173654 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181864525 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80098052 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚316679714610462 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 % _timestamp‚1.7884927999919012e+09÷Ά1†2é- _wandbruntime‚4899  _runtime‚4899_~{;bêÜ 1loss/FBt4/assignment_loss‚0.14087585356047375 2loss/SRqqt1/assignment_loss‚1.7033623952938373 0loss/SRqqt1/detection_loss‚2.083087831042534 /loss/FBt2/detection_loss‚2.6081691318088107 $loss/total_loss‚41.0161400462962 0loss/SRqqt2/detection_loss‚2.158939566682367 1loss/SRqqt4/detection_loss‚1.0619673561844705 epoch‚3 1loss/SRqqt3/detection_loss‚1.3060080945806605 0loss/FRt1/assignment_loss‚3.2335585258046136 0loss/FRt4/assignment_loss‚1.5519478902930306 trainer/global_step‚4599 2loss/SRqqt3/assignment_loss‚1.2937152036268946 2loss/FBt2/assignment_loss‚0.062479718406494716 3loss/FBt3/assignment_loss‚0.0008119381109281676 -loss/FRt3/detection_loss‚2.20178785371153 0loss/FBt4/detection_loss‚0.46319498544415605 _runtime‚4899.635068476 _step‚376 /loss/FRt2/assignment_loss‚3.397653325398763 2loss/FBt1/assignment_loss‚0.043012572442080454 /loss/FRt1/detection_loss‚2.2374330627136545 /loss/FRt4/detection_loss‚2.0477560209553864 /loss/FBt3/detection_loss‚2.6457217832406363 0loss/FRt3/assignment_loss‚2.0791761310477006 /loss/FRt2/detection_loss‚2.4540823320584417 /loss/FBt1/detection_loss‚3.4453162625432014 % _timestamp‚1.7884927999948099e+09 2loss/SRqqt2/assignment_loss‚1.7584910214166432 2loss/SRqqt4/assignment_loss‚1.0375911899288928qšogëá 0loss/FRt3/assignment_loss‚2.0791761310477006 /loss/FRt2/detection_loss‚2.4540823320584417 /loss/FBt1/detection_loss‚3.4453162625432014 % _timestamp‚1.7884927999948099e+09 2loss/SRqqt2/assignment_loss‚1.7584910214166432 2loss/SRqqt4/assignment_loss‚1.0375911899288928 1loss/FBt4/assignment_loss‚0.14087585356047375 2loss/SRqqt1/assignment_loss‚1.7033623952938373 0loss/SRqqt1/detection_loss‚2.083087831042534 /loss/FBt2/detection_loss‚2.6081691318088107 $loss/total_loss‚41.0161400462962 0loss/SRqqt2/detection_loss‚2.158939566682367 1loss/SRqqt4/detection_loss‚1.0619673561844705 epoch‚3 1loss/SRqqt3/detection_loss‚1.3060080945806605 0loss/FRt1/assignment_loss‚3.2335585258046136 0loss/FRt4/assignment_loss‚1.5519478902930306 trainer/global_step‚4599 2loss/SRqqt3/assignment_loss‚1.2937152036268946 2loss/FBt2/assignment_loss‚0.062479718406494716 3loss/FBt3/assignment_loss‚0.0008119381109281676 -loss/FRt3/detection_loss‚2.20178785371153 0loss/FBt4/detection_loss‚0.46319498544415605 _runtime‚4899.635068476 _step‚376 /loss/FRt2/assignment_loss‚3.397653325398763 2loss/FBt1/assignment_loss‚0.043012572442080454 /loss/FRt1/detection_loss‚2.2374330627136545 /loss/FRt4/detection_loss‚2.0477560209553864 /loss/FBt3/detection_loss‚2.6457217832406363ød>Ÿ’ì‚(:‡ …øèÔùÜã® gpu.0.gpu‚47.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;Uï‚í‚(:÷ …øèÔÛ disk.sda2.in‚0% disk./.usageGB‚1172.5475425720215 disk.dm-0.in‚ 20154.0390625 disk.dm-1.in‚0 disk.dm-1.out‚ 145.640625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.71484375 disk.sda.out‚ 145.640625 disk.sda2.out‚0 network.recv‚ 1.634242e+06) disk./.usagePercent‚78.69064941492653) disk.nvme0c0n1.out‚122526.90869140625 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 561.21484375 proc.cpu.threads‚17 network.sent‚ 1.0805094e+07$ disk.dm-0.out‚122526.90869140625 disk.sda.in‚0$ memory_percent‚96.57461643218994# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 cpu‚25.200396071349143# disk.nvme0c0n1.in‚ 20154.0390625! disk.nvme0n1.in‚ 20154.0390625' disk.nvme0n1.out‚122526.90869140625 disk.sda3.out‚ 145.640625mÕ2š’î‚(:‡ ”øèÔãî ¯ gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.831' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f;µ‹ï‚(:€ ”øèÔ’½þ² proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.71484375) disk.nvme0c0n1.out‚122551.20947265625" disk.nvme0n1.in‚20274.65234375' disk.nvme0n1.out‚122551.20947265625 disk.dm-0.in‚20274.65234375$ disk.dm-0.out‚122551.20947265625 disk.sda.out‚ 146.04296875 disk.sda1.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚20274.65234375$ memory_percent‚96.57471179962158( proc.memory.availableMB‚ 561.19921875* proc.memory.percent‚17.908453941345215 cpu‚25.200064731078374 disk.sda1.out‚0 network.sent‚ 1.0913788e+07 network.recv‚ 1.641018e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 146.04296875 disk.sda.in‚0) disk./.usagePercent‚78.72433795837227% disk./.usageGB‚1173.0495262145996 disk.sda3.out‚ 146.04296875 disk.sda2.in‚0# proc.memory.rssMB‚ 2934.12109375¹¼úð‚(:’ £øèԈ߯ gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.554' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.110800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ꮃñ‚(:ø £øèÔ»ºâ¯ disk.dm-1.in‚0 disk.dm-1.out‚ 146.55859375 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.71875$ memory_percent‚96.57316207885742& proc.memory.availableMB‚ 561.453125* proc.memory.percent‚17.908453941345215 cpu‚25.1997491049187 proc.cpu.threads‚17 network.recv‚ 1.643929e+06$ disk.nvme0c0n1.in‚20274.65234375" disk.nvme0n1.in‚20274.65234375 disk.dm-0.in‚20274.65234375$ disk.dm-0.out‚122722.29345703125# proc.memory.rssMB‚ 2934.12109375 network.sent‚ 1.0918127e+07( disk./.usagePercent‚78.7163574355074 disk.sda1.in‚0 disk.sda3.out‚ 146.55859375 disk.sda.in‚0 disk.sda.out‚ 146.55859375% disk./.usageGB‚1172.9306106567383 disk.sda1.out‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚122722.29345703125' disk.nvme0n1.out‚122722.29345703125 disk.sda3.in‚0€ÓE2ò- _wandbruntime‚4946  _runtime‚4946Š6?¦ÆóÀ _step‚377 & lr-AdamW/pg1‚0.0006124056925435025 & lr-AdamW/pg2‚0.0006124056925435025 trainer/global_step‚4649 $ _timestamp‚1.788492846908825e+09 _runtime‚4946.549207924Ý“DûËôÅ & lr-AdamW/pg2‚0.0006124056925435025 trainer/global_step‚4649 $ _timestamp‚1.788492846908825e+09 _runtime‚4946.549207924 _step‚377 & lr-AdamW/pg1‚0.0006124056925435025ù{/2õ- _wandbruntime‚4946  _runtime‚4946À“6$öðO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359267172059136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4761803154432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102856680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72116560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80924880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚364235041069568 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚315221365306576 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328180321897984 % _timestamp‚1.7884928469099188e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚359267148119040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚323419291388416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25739259 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80924878 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4761030509568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102833664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4761030509568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364234820890112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚323419309391360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4743886413376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328180321897984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4967672771072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 WJDeviceStatsMonitor.on_train_batc’Š Àh_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319965251719952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80924878 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46377835 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80924880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 _runtime‚4946.550354515 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319966039407724‰ËÍû'÷õO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4744657061788 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328180321897984 % _timestamp‚1.7884928469099188e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚359267148119040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚323419291388416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25739259 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80924878 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4761030509568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102833664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4761030509568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364234820890112 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚323419309391360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4743886413376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328180321897984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4967672771072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319965251719952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80924878 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46377835 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80924880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 _runtime‚4946.550354515 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319966039407724 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359267172059136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4761803154432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102856680 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72116560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80924880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚364235041069568 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚315221365306576 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4968923434496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚315287448781696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25744408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4745658201874 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102855375 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚323486991327744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183820171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183820171 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4744887553254 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72131373 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183797127 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80941754 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚359342520401920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚359342496461824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚315287431742336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72131893 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25743890 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183797127 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚323486991327744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46387485 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚379 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46387483 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4762035255296 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚328249026583040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102878417 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4968731402752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4947.441814083 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚518 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102878417 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80941752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364311227864576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884928478009543e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚328249817244160tì¸0'úÿM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4968731402752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4947.441814083 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚518 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102878417 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80941752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364311227864576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884928478009543e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚328249817244160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4968923434496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚315287448781696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25744408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4745658201874 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102855375 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚323486991327744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183820171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183820171 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4744887553254 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72131373 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183797127 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80941754 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚359342520401920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚359342496461824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚315287431742336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72131893 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25743890 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183797127 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚323486991327744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46387485 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚379 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46387483 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4762035255296 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚328249026583040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102878417 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042û]šš.2û- _wandbruntime‚4947  _runtime‚4947?:Æ\üÖ 1loss/SRqqt1/detection_loss‚2.5137891461772304 0loss/FRt2/assignment_loss‚3.0626560228842274 /loss/FRt4/assignment_loss‚1.513503754625515 2loss/SRqqt3/assignment_loss‚1.2938188409289488 /loss/FRt1/detection_loss‚2.4188267100188465 1loss/SRqqt3/detection_loss‚1.3845382469777878 /loss/FBt2/detection_loss‚2.3458075672388077 -loss/FBt1/detection_loss‚3.95509997010231 _step‚380 1loss/SRqqt2/assignment_loss‚1.825864373195556 % _timestamp‚1.7884928478026369e+09 /loss/FRt3/detection_loss‚2.4520046344170203 1loss/SRqqt2/detection_loss‚2.3583563239343706 /loss/FBt3/detection_loss‚2.1247212886810303 &loss/total_loss‚41.697146291445385 /loss/FRt1/assignment_loss‚3.081002689070172 2loss/SRqqt4/assignment_loss‚1.0146304159488988 1loss/FBt1/assignment_loss‚0.19791538788376783 0loss/FBt2/assignment_loss‚0.1939685182805988 0loss/FBt4/assignment_loss‚0.0824682978789042 1loss/SRqqt4/detection_loss‚0.8077074500591788 2loss/SRqqt1/assignment_loss‚1.6437050070493453 /loss/FBt4/detection_loss‚0.4005645686461602 epoch‚3 trainer/global_step‚4649 _runtime‚4947.443077865 /loss/FRt2/detection_loss‚2.9459701975186667 0loss/FRt3/assignment_loss‚2.5370100094721866 0loss/FBt3/assignment_loss‚0.2106265354261268 /loss/FRt4/detection_loss‚1.3325903350297286ú<‘ðHýÛ /loss/FRt1/assignment_loss‚3.081002689070172 2loss/SRqqt4/a°¹Ûlssignment_loss‚1.0146304159488988 1loss/FBt1/assignment_loss‚0.19791538788376783 0loss/FBt2/assignment_loss‚0.1939685182805988 0loss/FBt4/assignment_loss‚0.0824682978789042 1loss/SRqqt4/detection_loss‚0.8077074500591788 2loss/SRqqt1/assignment_loss‚1.6437050070493453 /loss/FBt4/detection_loss‚0.4005645686461602 epoch‚3 trainer/global_step‚4649 _runtime‚4947.443077865 /loss/FRt2/detection_loss‚2.9459701975186667 0loss/FRt3/assignment_loss‚2.5370100094721866 0loss/FBt3/assignment_loss‚0.2106265354261268 /loss/FRt4/detection_loss‚1.3325903350297286 1loss/SRqqt1/detection_loss‚2.5137891461772304 0loss/FRt2/assignment_loss‚3.0626560228842274 /loss/FRt4/assignment_loss‚1.513503754625515 2loss/SRqqt3/assignment_loss‚1.2938188409289488 /loss/FRt1/detection_loss‚2.4188267100188465 1loss/SRqqt3/detection_loss‚1.3845382469777878 /loss/FBt2/detection_loss‚2.3458075672388077 -loss/FBt1/detection_loss‚3.95509997010231 _step‚380 1loss/SRqqt2/assignment_loss‚1.825864373195556 % _timestamp‚1.7884928478026369e+09 /loss/FRt3/detection_loss‚2.4520046344170203 1loss/SRqqt2/detection_loss‚2.3583563239343706 /loss/FBt3/detection_loss‚2.1247212886810303 &loss/total_loss‚41.697146291445385ü娒þ‚(:‡ ²øèÔ豊± gpu.0.gpu‚34.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T…szƒÿ‚(:ø ²øèÔޜմ' disk.nvme0n1.out‚123102.07080078125 cpu‚25.19945065104639 proc.cpu.threads‚17 network.sent‚ 1.0923068e+07 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 147.16796875 disk.sda.in‚0) disk.nvme0c0n1.out‚123102.07080078125" disk.nvme0n1.in‚20274.65234375% proc.memory.availableMB‚ 561.15625 network.recv‚ 1.647378e+06) disk./.usagePercent‚78.72698354023112 disk.drbd1260.in‚0 disk.drbd1260.out‚0.71875 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚20274.65234375 disk.dm-0.in‚20274.65234375% disk./.usageGB‚1173.0889472961426 disk.sda2.in‚0$ disk.dm-0.out‚123102.07080078125# memory_percent‚96.5749740600586# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 disk.sda3.out‚ 147.16796875 disk.sda.out‚ 147.16796875 disk.sda2.out‚0}‡öO’€‚(:‡ ÁøèÔ–¬ø® gpu.0.gpu‚41.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚127.469' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.4938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€fg‡‚(:ü ÁøèÔúΧ°" disk.nvme0n1.in‚20274.65234375$ disk.dm-0.out‚123536.40673828125$ memory_percent‚96.57366275787354 disk.sda3.out‚ 147.6796875 disk.sda1.out‚0) disk.nvme0c0n1.out‚123536.40673828125' disk.nvme0n1.out‚123536.40673828125 disk.dm-0.in‚20274.65234375# proc.memory.rssMB‚ 2934.12109375 cpu‚25.19919978707183 disk.sda.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 561.37109375* proc.memory.percent‚17.908453941345215 proc.cpu.threads‚17 network.sent‚ 1.1032759e+07 network.recv‚ 1.654909e+06 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 147.6796875 disk.drbd1260.out‚ 0.72265625 disk.sda1.in‚0 disk.sda2.in‚0) disk./.usagePercent‚78.72698763635167% disk./.usageGB‚1173.0890083312988 disk.drbd1260.in‚0 disk.sda.out‚ 147.6796875$ disk.nvme0c0n1.in‚20274.652343758#‚‚(:’ ÐøèÔÞ‹Ç® gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.524' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.904799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|7üჃ‚(:ø ÐøèÔµ À±" disk.nvme0c0n1.in‚ 20405.640625 disk.nvme0n1.in‚ 20405.640625 disk.dm-0.in‚ 20405.640625( proc.memory.availableMB‚ 561.11328125 proc.cpu.threads‚17 network.sent‚ 1.1038786e+07 disk.drbd1260.in‚0 disk.sda.out‚ 148.05078125) disk.nvme0c0n1.out‚123830.74658203125 network.recv‚ 1.659572e+06% disk./.usageGB‚1173.1168251037598 disk.dm-1.out‚ 148.05078125 disk.drbd1260.out‚ 0.72265625 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0n1.out‚123830.74658203125# memory_percent‚96.5752363204956 disk.sda3.in‚0 disk.sda3.out‚ 148.05078125 disk.dm-1.in‚0 disk.sda.in‚0$ disk.dm-0.out‚123830.74658203125# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 cpu‚25.19889345676614) disk./.usagePercent‚78.72885444329134 disk.sda2.in‚0 disk.sda2.out‚0‹ØF’„‚(:‡ ßøèÔž±Ü­ gpu.0.gpu‚37.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.713' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qŸ¥þ…‚(:ö ßøèÔÁ»å¯% disk./.usageGB‚1172.9522552490234 disk.drbd1260.in‚0 disk.sda.in‚0( disk.nvme0c0n1.out‚124098.3525390625 disk.dm-0.in‚ 20405.640625# disk.dm-0.out‚124098.3525390625$ memory_percent‚96.57366275787354 network.sent‚ 1.1044237e+07 network.recv‚ 1.663507e+06 disk.sda3.out‚ 148.55078125( proc.memory.availableMB‚ 561.37109375# proc.memory.rssMB‚ 2934.12109375 cpu‚25.198618733723993 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.7265625 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 20405.640625 disk.nvme0n1.in‚ 20405.640625* proc.memory.percent‚17.908453941345215 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 148.55078125 disk.sda.out‚ 148.55078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚124098.3525390625) disk./.usagePercent‚78.71781002225669Ujÿ¤2†- _wandbruntime‚4996  _runtime‚4996`RFXƇÀ & lr-AdamW/pg1‚0.0006075925062034444 & lr-AdamW/pg2‚0.0006075925062034444 trainer/global_step‚4699 $ _timestamp‚1.788492896500298e+09 _runtime‚4996.140644833 _step‚381#FéˈŠ& lr-AdamW/pg1‚0.0006075925062034444 & lr-AdamW/pg2‚0.0006075925062034444 trainer/global_step‚4699 $ _timestamp‚1.788492896500298e+09 _runtime‚4996.140644833 _step‚381ý°Ær2‰- _wandbruntime‚4996  _runtime‚4996¹™ö'ŠðO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323318644093060 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚326804288354816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚326804306357760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103922566 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81768580 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4811282209792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚363034589198336 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103945582 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚318524687094576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331616361212416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚326804306357760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72864930 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚363034565258240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4811282209792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5020363481600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5020165145088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚4699 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26004693 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚368054952679936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323319431780832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81768578 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103945582 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81768578 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81768580 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚382 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 _runtime‚4996.141874169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4812054854656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 % _timestamp‚1.7884928965012438e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331615570564608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚331616361212416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103922566 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚368054730403328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4812054854656 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331615570564608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚331616361212416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103922566 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚368054730403328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4812054854656 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚326804306357760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103922566 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81768580 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4811282209792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚363034589198336 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103945582 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328þ,_ÙÀ2Œ- _wandbruntime‚4996  _runtime‚4996Õ€ l÷ùM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81785454 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26009421 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚368131150551552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _runtime‚ 4996.99086933 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103967319 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103967319 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5021236950528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚318590753530336 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚331685065910784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323385711668698 _step‚383 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81785452 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185752773 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81785452 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4795728786982 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81785454 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72879304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207583232 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185752773 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323386499356678 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4794958138362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚363109937541120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516]z+'ŽþM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185752773 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323386499356678 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4794958138362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚363109937541120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81785454 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26009421 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚368131150551552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _runtime‚ 4996.99086933 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103967319 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103967319 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5021236950528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚318590753530336 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚331685065910784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323385711668698 _step‚383 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81785452 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185752773 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81785452 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4795728786982 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81785454 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72879304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207583232 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ÿX(@Š2- _wandbruntime‚4996  _runtime‚4996°Pó?`Ú /loss/FRt2/assignment_loss‚3.515944517321057 1loss/FBt2/assignment_loss‚0.37743389601184224 1loss/SRqqt3/detection_loss‚1.1617377478895443 /loss/FBt3/detection_loss‚2.8921402295430503 /loss/FRt3/assignment_loss‚2.595621290255566 /loss/FRt4/detection_loss‚1.7146690306594459 1loss/SRqqt4/detection_loss‚1.1146425516573621 /loss/FBt4/detection_loss‚0.5551783917079041 2loss/SRqqt1/assignment_loss‚1.6479915196282997 /loss/FRt2/detection_loss‚2.2854904524853787 /loss/FRt3/detection_loss‚2.3036053252159334 0loss/FRt1/assignment_loss‚3.1265674840945463 4loss/FBt3/assignment_loss‚0.00022888795422204566 1loss/SRqqt1/detection_loss‚2.0742374236385026 &loss/total_loss‚41.835921540857306 epoch‚3 _runtime‚4996.992428659 _step‚384 # _timestamp‚1.78849289735216e+09 2loss/SRqqt3/assignment_loss‚1.5288766219413705 1loss/FBt4/assignment_loss‚0.08025379278003159 .loss/FRt1/detection_loss‚2.228819608115233 1loss/SRqqt2/detection_loss‚2.4482642505317926 /loss/FBt1/detection_loss‚2.4421312625591574 /loss/FBt2/detection_loss‚2.6855608983473345 2loss/SRqqt2/assignment_loss‚1.8664561295881867 1loss/SRqqt4/assignment_loss‚1.138025088632693 trainer/global_step‚4699 /loss/FRt4/assignment_loss‚1.689607775792843 1loss/FBt1/assignment_loss‚0.36243736450600966;i…­e‘ß 1loss/FBt1/assignment_loss‚0.36243736450600966 /loss/FRt2/assignment_loss‚3.515944517321057 1loss/FBt2/assignment_loss‚0.37743389601184224 1loss/SRqqt3/detection_loss‚1.1617377478895443 /loss/FBt3/detection_loss‚2.8921402295430503 /loss/FRt3/assignment_loss‚2.595621290255566 /loss/FRt4/detection_loss‚1.7146690306594459 1loss/SRqqt4/detection_loss‚1.1146425516573621 /loss/FBt4/detection_loss‚0.5551783917079041 2loss/SRqqt1/assignment_loss‚1.6479915196282997 /loss/FRt2/detection_loss‚2.2854904524853787 /loss/FRt3/detection_loss‚2.3036053252159334 0loss/FRt1/assignment_loss‚3.1265674840945463 4loss/FBt3/assignment_loss‚0.00022888795422204566 1loss/SRqqt1/detection_loss‚2.0742374236385026 &loss/total_loss‚41.835921540857306 epoch‚3 _runtime‚4996.992428659 _step‚384 # _timestamp‚1.78849289735216e+09 2loss/SRqqt3/assignment_loss‚1.5288766219413705 1loss/FBt4/assignment_loss‚0.08025379278003159 .loss/FRt1/detection_loss‚2.228819608115233 1loss/SRqqt2/detection_loss‚2.4482642505317926 /loss/FBt1/detection_loss‚2.4421312625591574 /loss/FBt2/detection_loss‚2.6855608983473345 2loss/SRqqt2/assignment_loss‚1.8664561295881867 1loss/SRqqt4/assignment_loss‚1.138025088632693 trainer/global_step‚4699 /loss/FRt4/assignment_loss‚1.689607775792843€+CÏ4œ’‚(:‘ îøèÔîÉ„² gpu.0.gpu‚50.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.656000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\G§~“‚(:ó îøèÔ ·¶ disk.dm-1.out‚ 149.23046875 disk.sda2.out‚0# proc.memory.rssMB‚ 2934.12109375 disk.nvme0n1.in‚ 20405.640625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚124152.7353515625* proc.memory.percent‚17.908453941345215 network.recv‚ 1.666951e+06& disk.nvme0n1.out‚124152.7353515625# disk.dm-0.out‚124152.7353515625 disk.sda3.out‚ 149.23046875 disk.sda.out‚ 149.23046875 disk.sda2.in‚0 cpu‚25.198361539725568 proc.cpu.threads‚17 network.sent‚ 1.1049298e+07' disk./.usagePercent‚78.726182748664 disk.sda3.in‚0 disk.drbd1260.out‚ 0.7265625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 20405.640625$ memory_percent‚96.57387733459473' proc.memory.availableMB‚ 561.3359375% disk./.usageGB‚1173.0770149230957 disk.dm-0.in‚ 20405.640625èy£’”‚(:‡ ýøèÔñ£‘® gpu.0.gpu‚30.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335050819231232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105034484 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚366802006337536 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335050819231232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82612280 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321828025921936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4844027583592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26270850 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5072639105024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335051609879040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4862306554880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4861533910016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73613485 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105011468 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335051609879040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26271387 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5072833247232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5045.808759829 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚330189285321216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4862306554880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚371874621502464 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 % _timestamp‚1.7884929461680405e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47342635 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187646764 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326672036466168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚330189285321216 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105011468 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4861533910016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73614024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚366801982397440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 _step‚386 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47342637 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371874839584768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187623746 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚330189303324160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 trainer/global_step‚4749 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0°à sõO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 H>DeviceStatsMonitor.on_trai‰Æ±6ˆ&n_batch_start/active.small_pool.peak‚27846 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335050819231232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105034484 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚366802006337536 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335050819231232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82612280 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321828025921936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4844027583592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26270850 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5072639105024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335051609879040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4862306554880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4861533910016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73613485 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105011468 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335051609879040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26271387 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5072833247232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5045.808759829 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚330189285321216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4862306554880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚371874621502464 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 % _timestamp‚1.7884929461680405e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47342635 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187646764 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326672036466168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚330189285321216 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105011468 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4861533910016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73614024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚366801982397440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 _step‚386 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47342637 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371874839584768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187623746 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚330189303324160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 trainer/global_step‚4749 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82612278 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321828008882576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326672824153940 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4844798232004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82612280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82612278 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335120314577408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4845028723470 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82629154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73628429 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47352285 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26276146 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321894092357696 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335119523916288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371951232041984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4845799372090 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82629152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4862538655744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5073681135616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4749 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4863311313920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82629152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105033179 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚330256985260544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187685375 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105056221 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187685375 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5073877361664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335120314577408 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187662331 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105056221 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105033179 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326739104041806 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4863311313920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187662331 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 _runtime‚5046.927808974 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4863311313920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187662331 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 _runtime‚5046.927808974 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335120314577408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4845028723470 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82629154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73628429 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47352285 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26276146 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321894092357696 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335119523916288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371951232041984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4845799372090 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82629152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4862538655744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5073681135616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4749 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4863311313920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82629152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105033179 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚330256985260544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187685375 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105056221 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187685375 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5073877361664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335120314577408 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187662331 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105056221 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105033179ƒõ‘Ë2¡- _wandbruntime‚5046  _runtime‚5046ÞÔa^¢Ø /loss/FRt4/detection_loss‚1.8206054993040328 %loss/total_loss‚41.28990363738867 _runtime‚5046.929750042 /loss/FRt2/detection_loss‚2.7294193223051058 2loss/SRqqt2/assignment_loss‚1.5646050202057642 _step‚388 .loss/FRt1/detection_loss‚2.643827799707651 /loss/FBt1/detection_loss‚2.3916974385579426 /loss/FBt2/detection_loss‚2.6523186401887373 1loss/FBt2/assignment_loss‚0.12224872725214003 trainer/global_step‚4749 2loss/SRqqt1/assignment_loss‚1.2334852707458706 1loss/SRqqt3/detection_loss‚1.6449077733434163 /loss/FBt4/detection_loss‚0.6117344971172252 1loss/FBt4/assignment_loss‚0.11732654433358697 /loss/FRt3/assignment_loss‚2.443332973364237 0loss/FRt4/assignment_loss‚1.7090692615934782 2loss/SRqqt4/assignment_loss‚1.0903921645242147 % _timestamp‚1.7884929472895052e+09 /loss/FRt2/assignment_loss‚3.324272403717041 0loss/FBt1/assignment_loss‚0.3191428856924176 0loss/SRqqt1/detection_loss‚2.121682321153036 1loss/SRqqt2/detection_loss‚2.1886384381967434 /loss/FBt3/detection_loss‚2.8641741360936845 0loss/FRt1/assignment_loss‚2.7998121705922214 2loss/SRqqt3/assignment_loss‚1.3353133944364695 2loss/FBt3/assignment_loss‚0.052960517072019035 .loss/FRt3/detection_loss‚2.502260364290025 1loss/SRqqt4/detection_loss‚1.0066760736016127 epoch‚3ïì]DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375694383208960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333574300290560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚370569423476736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333574300290560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189579366 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚325131330670576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338486858545664 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74361792 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106100370 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83455978 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4912558255104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83455980 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74362317 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚338486067897856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 # _timestamp‚1.78849299615142e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83455978 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26536757 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚375694601291264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5125177814528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚325131347709936 trainer/global_step‚4799 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106123386 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4894098168700 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 _runtime‚5095.792197401 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4912558255104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330026216527048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4911785610240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189556348 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106123386 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26537280 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338486067897856 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106100370 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664SÜSù'¯óO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4894868817112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚338486858545664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚370569399536640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚390 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375694383208960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333574300290560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚370569423476736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333574300290560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189579366 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚325131330670576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338486858545664 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74361792 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106100370 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83455978 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4912558255104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83455980 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74362317 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚338486067897856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 # _timestamp‚1.78849299615142e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83455978 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26536757 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚375694601291264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5125177814528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚325131347709936 trainer/global_step‚4799 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106123386 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4894098168700 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 _runtime‚5095.792197401 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4912558255104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330026216527048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4911785610240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189556348 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106123386 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26537280 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338486067897856 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106100370 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5124983672320 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333574282287616†… Á2°- _wandbruntime‚5096  _runtime‚5096–æØM±ùM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189617977 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230651904 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370644771819520 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚325197414145696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4913563014144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74377261 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338555563244032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106145123 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106122081 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338554772582912 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375770763213312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333642000229888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4913563014144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74376721 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206711808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5126015333888 % _timestamp‚1.7884929971354623e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26542576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4912790355968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83472852 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚325197397106336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚338555563244032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4912790355968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5126222045696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4895869957198 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83472854 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333642000229888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47834685 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106122081 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83472854 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytesÿòW².large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333641982226944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375770993865216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333641982226944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106145123 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26542038 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189594933 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330092496414914 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _runtime‚ 5096.77686987ÞÏ£@'²þM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375770993865216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333641982226944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106145123 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26542038 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189594933 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330092496414914 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _runtime‚ 5096.77686987 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189617977 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230651904 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370644771819520 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚325197414145696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4913563014144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74377261 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338555563244032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106145123 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106122081 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338554772582912 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375770763213312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333642000229888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4913563014144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74376721 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206711808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5126015333888 % _timestamp‚1.7884929971354623e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26542576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4912790355968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83472852 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚325197397106336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚338555563244032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4912790355968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5126222045696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4895869957198 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83472854 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333642000229888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47834685 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106122081 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83472854 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333641982226944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664‡ñe‹2³- _wandbruntime‚5096  _runtime‚5096z¸ª»]´× epoch‚3 1loss/SRqqt1/detection_loss‚2.5081228399276734 1loss/SRqqt3/detection_loss‚1.4388317850307892 0loss/SRqqt4/detection_loss‚1.141951501764583 _runtime‚5096.779211833 /loss/FRt4/detection_loss‚1.6815077397776277 /loss/FRt2/assignment_loss‚3.223964700193116 /loss/FRt3/assignment_loss‚2.382539168292401 0loss/FBt4/assignment_loss‚0.1257185419499853 /loss/FBt4/detection_loss‚0.5463521442540428 &loss/total_loss‚41.406957326908866 _step‚392 0loss/FRt4/assignment_loss‚1.2204339472871077 /loss/FRt2/detection_loss‚2.4786554097467173 /loss/FBt2/detection_loss‚2.8429198794894748 % _timestamp‚1.7884929971388223e+09 0loss/FRt1/assignment_loss‚3.4696984334689813 2loss/SRqqt1/assignment_loss‚1.3384418570995331 /loss/FRt1/detection_loss‚2.3974494919544314 0loss/SRqqt2/detection_loss‚2.425615626162496 1loss/SRqqt2/assignment_loss‚1.666985451918224 1loss/SRqqt3/assignment_loss‚1.218944594832295 2loss/SRqqt4/assignment_loss‚0.7830360347800183 0loss/FBt1/assignment_loss‚0.3403176932934002 /loss/FRt3/detection_loss‚2.2714517958023968 /loss/FBt1/detection_loss‚2.8224496878683567 /loss/FBt3/detection_loss‚2.8658453713763845 trainer/global_step‚4799 0loss/FBt2/assignment_loss‚0.2107706032693386 3loss/FBt3/assignment_loss‚0.0049530273694968355þû bµÜ /loss/FRt2/assignment_loss‚3.223964700193116 /loss/FRt3/assignment_loss‚2.382539168292401 0loss/FBt4/assignment_loss‚0.1257185419499853 /loss/FBt4/detection_loss‚0.5463521442540428 &loss/total_loss‚41.406957326908866 _step‚392 0loss/FRt4/assignment_loss‚1.2204339472871077 /loss/FRt2/detection_loss‚2.4786554097467173 /loss/FBt2/detection_loss‚2.8429198794894748 % _timestamp‚1.7884929971388223e+09 0loss/FRt1/assignment_loss‚3.4696984334689813 2loss/SRqqt1/assignment_loss‚1.3384418570995331 /loss/FRt1/detection_loss‚2.3974494919544314 0loss/SRqqt2/detection_loss‚2.425615626162496 1loss/SRqqt2/assignment_loss‚1.666985451918224 1loss/SRqqt3/assignment_loss‚1.218944594832295 2loss/SRqqt4/assignment_loss‚0.7830360347800183 0loss/FBt1/assignment_loss‚0.3403176932934002 /loss/FRt3/detection_loss‚2.2714517958023968 /loss/FBt1/detection_loss‚2.8224496878683567 /loss/FBt3/detection_loss‚2.8658453713763845 trainer/global_step‚4799 0loss/FBt2/assignment_loss‚0.2107706032693386 3loss/FBt3/assignment_loss‚0.0049530273694968355 epoch‚3 1loss/SRqqt1/detection_loss‚2.5081228399276734 1loss/SRqqt3/detection_loss‚1.4388317850307892 0loss/SRqqt4/detection_loss‚1.141951501764583 _runtime‚5096.779211833 /loss/FRt4/detection_loss‚1.6815077397776277ˆ˜D’¶‚(:‡ ÈùèÔÇéÔ± gpu.0.gpu‚42.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.928' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1856 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0… Åf·‚(:ö ÈùèÔâç³ disk.drbd1260.out‚ 0.7421875$ memory_percent‚96.57459259033203 cpu‚25.196682090778268% disk./.usageGB‚1174.2205238342285' disk.nvme0n1.out‚126345.46142578125 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2934.12109375 proc.cpu.threads‚17 network.sent‚ 1.1289634e+07 network.recv‚ 1.697258e+06# disk.nvme0c0n1.in‚ 20730.5078125! disk.nvme0n1.in‚ 20730.5078125$ disk.dm-0.out‚126345.46142578125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.in‚0) disk./.usagePercent‚78.80292459115802) disk.nvme0c0n1.out‚126345.46142578125 disk.sda3.out‚ 151.7890625 disk.sda.out‚ 151.7890625 disk.sda2.in‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 561.21875* proc.memory.percent‚17.908453941345215 disk.dm-1.out‚ 151.7890625 disk.dm-0.in‚ 20730.5078125=pôŠŽ¸‚(:ƒ ×ùèÔÃÜÍ´ gpu.0.gpu‚30.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.3' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.86 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04+—ìv¹‚(:ë ×ùèÔŽíÿ¸! disk.nvme0n1.in‚ 20730.5078125 disk.dm-0.in‚ 20730.5078125 disk.sda3.in‚0# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 disk.sda.out‚152.0625 disk.sda1.in‚0& disk.nvme0n1.out‚126400.8154296875 disk.dm-1.out‚152.0625$ memory_percent‚96.57306671142578% proc.memory.availableMB‚ 561.46875 cpu‚25.196440094722846 network.recv‚ 1.702848e+06) disk./.usagePercent‚78.80292459115802 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚126400.8154296875 proc.cpu.threads‚17 network.sent‚ 1.1397621e+07 disk.drbd1260.out‚ 0.74609375 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚126400.8154296875 disk.sda3.out‚152.0625 disk.dm-1.in‚0% disk./.usageGB‚1174.2205238342285 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 20730.5078125$¥‚º‚(:’ æùèÔŸŸŠ¯ gpu.0.gpu‚38.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.022399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À,ðc‡»‚(:ü æùèÔ¨ô–³ disk.dm-1.out‚ 152.48828125 disk.drbd1260.out‚ 0.74609375 disk.sda.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 20730.5078125! disk.nvme0n1.in‚ 20730.5078125* proc.memory.percent‚17.908453941345215 network.recv‚ 1.706303e+06) disk./.usagePercent‚78.80291870298473 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚126744.82568359375' disk.nvme0n1.out‚126744.82568359375% disk./.usageGB‚1174.2204360961914 disk.sda3.out‚ 152.48828125 disk.drbd1260.in‚0 disk.sda.out‚ 152.48828125 disk.dm-0.in‚ 20730.5078125$ disk.dm-0.out‚126744.82568359375$ memory_percent‚96.57301902770996# proc.memory.rssMB‚ 2934.12109375 network.sent‚ 1.1402764e+07 disk.sda3.in‚0' proc.memory.availableMB‚ 561.4765625 cpu‚25.196173794621032 proc.cpu.threads‚17Òüñ?¼‚(:’ õùèÔâœô­ gpu.0.gpu‚51.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚140.227' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.045399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶_ €½‚(:õ õùèÔ§æ± disk.sda2.out‚0* proc.memory.percent‚17.908453941345215 network.sent‚ 1.141082e+07) disk./.usagePercent‚78.80973003944072) disk.nvme0c0n1.out‚127214.31103515625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 561.7265625 disk.sda3.in‚0 disk.sda3.out‚ 152.85546875 disk.dm-1.out‚ 152.85546875 disk.drbd1260.in‚0 disk.sda.out‚ 152.85546875$ memory_percent‚96.57149314880371# proc.memory.rssMB‚ 2934.12109375% disk./.usageGB‚1174.3219299316406# disk.nvme0c0n1.in‚ 20730.5078125! disk.nvme0n1.in‚ 20730.5078125 cpu‚25.209265587024632 proc.cpu.threads‚17 network.recv‚ 1.711175e+06' disk.nvme0n1.out‚127214.31103515625 disk.dm-0.in‚ 20730.5078125$ disk.dm-0.out‚127214.31103515625 disk.drbd1260.out‚0.75 disk.sda1.out‚0Ї(鑾‚(:† „úèÔôÇø­ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚113.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aàõ¬¿‚(:ô „úèÔâÔè± network.sent‚ 1.141679e+07! disk.nvme0n1.in‚ 20863.4765625 disk.dm-1.out‚ 153.21484375 disk.sda.in‚0 disk.sda.out‚ 153.21484375# proc.memory.rssMB‚ 2934.12109375 network.recv‚ 1.715364e+06 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 20863.4765625) disk.nvme0c0n1.out‚127605.08447265625 disk.dm-0.in‚ 20863.4765625 disk.dm-1.in‚0$ memory_percent‚96.57607078552246 cpu‚25.223498163291342( disk./.usagePercent‚78.8189061174905% disk./.usageGB‚1174.4586601257324 disk.sda2.in‚0$ disk.dm-0.out‚127605.08447265625 disk.sda3.in‚0 disk.drbd1260.in‚0' proc.memory.availableMB‚ 560.9765625 proc.cpu.threads‚17' disk.nvme0n1.out‚127605.08447265625 disk.sda3.out‚ 153.21484375 disk.drbd1260.out‚0.75 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.908453941345215¡p)œÀ‚(:‘ “úèÔÀꎯ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.975' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.794999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ƒ&ÀyÁ‚(:î “úèÔ±´ð±$ memory_percent‚96.59419059753418' proc.memory.availableMB‚ 558.0078125 disk.drbd1260.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2934.12109375 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 153.6484375( disk./.usagePercent‚78.8365386364113& disk.nvme0c0n1.out‚127979.14453125$ disk.nvme0n1.out‚127979.14453125 disk.dm-0.in‚ 20863.4765625! disk.dm-0.out‚127979.14453125 disk.sda3.in‚0 network.sent‚ 1.142193e+07 network.recv‚ 1.718982e+06% disk./.usageGB‚1174.7213973999023# disk.nvme0c0n1.in‚ 20863.4765625 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.908453941345215 cpu‚25.238129828974362! disk.nvme0n1.in‚ 20863.4765625 disk.sda3.out‚ 153.6484375 disk.drbd1260.out‚ 0.75390625 disk.sda.out‚ 153.6484375 disk.sda1.out‚0 disk.sda2.out‚0]¦æœÂ‚(:‘ ¢úèÔÌæÊ® gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.674799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÒyÂ(:î ¢úèÔþ°»³ disk.dm-1.in‚0 disk.drbd1260.out‚ 0.75390625 disk.sda.out‚ 154.14453125 disk.sda1.out‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚128375.95703125! disk.nvme0n1.in‚ 20863.4765625 disk.dm-0.in‚ 20863.4765625" proc.memory.availableMB‚561.25# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 proc.cpu.threads‚17 disk.sda3.out‚ 154.14453125 disk.dm-1.out‚ 154.14453125 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 20863.4765625! disk.dm-0.out‚128375.95703125$ memory_percent‚96.57440185546875) disk./.usagePercent‚78.83635559102432 disk.sda2.out‚0$ disk.nvme0n1.out‚128375.95703125 cpu‚25.253707162114882 network.sent‚ 1.1424984e+07 network.recv‚ 1.722344e+06% disk./.usageGB‚1174.7186698913574 disk.sda3.in‚0ì‹—2Ä- _wandbruntime‚5190  _runtime‚5190Çj•s!Å×T XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 192031904 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18217 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚48573033 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1005 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚343279060877312 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚343279060877312 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578072504 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5186722186240 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10558147584 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28115 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634825 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4971948872192 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚48573031 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 192050121 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 192031904 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 107315960 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚75558773 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884930910540946e+09 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4954051156942 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 107334175 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚334602336115342 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚338307112005120 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 580044288 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚329648284958400 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 580044288 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚84715944 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 11892004352 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 192050121 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚39 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1005 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚84715944 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚184 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 375175680 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18217 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4972528916480 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 107334175 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚343279658924544 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4972528916480 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚380846094354944 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚375659217338368 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18215 aTDeEÍà[ viceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 5186853076480 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚516 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚338307130008064 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚26985740 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚182 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1005 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚380845939524608 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 130890240 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚334602931227206 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚4835 _step‚393 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4971948872192 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3125 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27846 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚84715946 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚329648301997760 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 107315960 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚338307130008064 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12155092992ÀI¤þb*ÆÜT eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚338307130008064 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚26985740 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚182 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1005 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚380845939524608 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 130890240 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚334602931227206 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚4835 _step‚393 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4971948872192 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3125 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27846 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚84715946 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚329648301997760 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 107315960 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚338307130008064 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 192031904 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18217 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚48573033 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1005 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚343279060877312 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚343279060877312 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578072504 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5186722186240 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10558147584 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28115 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634825 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4971948872192 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚48573031 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 192050121 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 192031904 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 107315960 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚75558773 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884930910540946e+09 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4954051156942 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 107334175 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚334602336115342 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚338307112005120 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 580044288 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚329648284958400 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 580044288 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚84715944 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 11892004352 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 192050121 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚39 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1005 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚84715944 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚184 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 375175680 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18217 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4972528916480 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 107334175 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚343279658924544 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4972528916480 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚380846094354944 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚375659217338368 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18215 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 5186853076480 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚516‰8ëÔù2Ç- _wandbruntime‚5192  _runtime‚5192v©n*f)ÈàR ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27846 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 578072504 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12155092992 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1005 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚343325591672320 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892004352 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 130890240 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27846 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚183 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 trainer/global_step‚4835 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚75571789 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚26990593 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3125 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚48581013 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 4972993665024 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚338352580004352 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚375706195483648 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18215 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 192071387 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚185 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4954514317906 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 192053170 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚26990776 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚84728918 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 4972413620736 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚338352598007296 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595111864 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 598047232 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 192053170 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚343324993625088 % _timestamp‚1.7884930924195278e+09 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 11518882825 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚329690580798496 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚84728916 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚516 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 107342469 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚375706219423744 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚380893382418432 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚329690597837856 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11129585664 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 11892004352 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5187186934784 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1005 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1005 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4972413620736 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3125 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 107324254 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚343325591672320 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 598047232 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚343324993625088 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18217 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 107324254 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 375175680 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12155092992 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28115 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚516 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚75571604 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 154830336 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4955092390410 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18215 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 5187317825024 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 10558147584 _runtime‚ 5192.06026886 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580044288zˆ"ÉåR ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚516 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 107342469 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚375706219423744 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚380893382418432 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚329690597837856 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11129585664 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 11892004352 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5187186934784 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1005 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1005 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4972413620736 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3125 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 107324254 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚343325591672320 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 598047232 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚343324993625088 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18217 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 107324254 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 375175680 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12155092992 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28115 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚516 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚75571604 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 154830336 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4955092390410 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18215 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 5187317825024 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 10558147584 _runtime‚ 5192.06026886 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580044288 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27846 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 578072504 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12155092992 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1005 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚343325591672320 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892004352 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 130890240 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27846 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚183 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 trainer/global_step‚4835 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚75571789 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚26990593 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3125 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚48581013 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 4972993665024 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚338352580004352 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚375706195483648 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18215 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 192071387 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚185 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4954514317906 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 192053170 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚26990776 ]ODeviceStatsMonitor.on_vali¦n›¢ãdation_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚84728918 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 4972413620736 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚338352598007296 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595111864 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 598047232 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 192053170 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚343324993625088 % _timestamp‚1.7884930924195278e+09 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 11518882825 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚329690580798496 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚84728916 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0Š1W‹¼‘Ê‚(:† ±úèÔ㦤­ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚33.0 gpu.0.powerWatts‚114.994' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JDr~Ë‚(:ó ±úèÔÑçÙ°# disk.dm-0.out‚128646.9892578125$ memory_percent‚96.57464027404785 network.recv‚ 1.72717e+06% disk./.usageGB‚1174.2472190856934 disk.sda2.in‚0' proc.memory.availableMB‚ 561.2109375# proc.memory.rssMB‚ 2934.12109375* proc.memory.percent‚17.908453941345215 network.sent‚ 1.1444433e+07 disk.dm-1.out‚ 154.6484375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚128646.9892578125! disk.nvme0n1.in‚ 20863.4765625 proc.cpu.threads‚17 disk.sda3.out‚ 154.6484375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 20863.4765625 cpu‚25.26748857971975) disk./.usagePercent‚78.80471613188263 disk.sda3.in‚0 disk.drbd1260.out‚ 0.7578125 disk.sda.out‚ 154.6484375# disk.nvme0c0n1.in‚ 20863.4765625& disk.nvme0n1.out‚128646.9892578125z÷ž2Ì- _wandbruntime‚5207  _runtime‚5207´“v¹sÍǦ 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.968849241733551 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.018801964819431305 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3477678596973419 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.45671144127845764 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 jet/accuracy_7_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.38376134634017944 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04063556343317032 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2653830051422119 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2322048544883728 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ( Purity/3FRt4SRqqt*FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2581247389316559 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15227364003658295 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06984104961156845 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5744665265083313 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9101654291152954 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2625969648361206 particle/accuracy_1_of_1‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_1_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.265325665473938 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4067467451095581 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6487901210784912 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3256944715976715 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08510638028383255 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03074757196009159 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.41130441427230835 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9892869591712952 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2648809552192688 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.36523836851119995 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.4166666865348816 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,jet/accuracy_4_of_5‚0.010618279688060284 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0219645444303751 jet/accuracy_3_of_7‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17874997854232788 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3184744119644165 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3645833432674408 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18434299528598785 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.018147870898246765 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.31998053193092346 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3523470163345337 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4945175349712372 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.4166666865348816 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2619466185569763 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.35960203409194946 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984994530677795 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.25823044776916504 +jet/accuracy_3_of_5‚0.08058755099773407 jet/accuracy_6_of_7‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5798439979553223 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2845926880836487 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28882548213005066 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 particle/accuracy_1_of_3‚1 ( Purity/3FRt4SRqqt0FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 trainer/global_step‚4835 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.535187304019928 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29611286520957947 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.43333330750465393 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333730697632 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 jet/accuracy_4_of_6‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2998655438423157 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3580247163772583 particle/accuracy_4_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5862839818000793 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2706093192100525 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.26541632413864136 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30297598242759705 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.36626681685447693 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9863973259925842 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025651821866631508 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17577949166297913 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.45621293783187866 -jet/accuracy_4_of_4‚0.0031902098562568426 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6033282279968262 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5357311964035034 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16258960962295532 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.202707439661026 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.40710073709487915 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25764936208724976 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.39430636167526245 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 'particle/f_score‚0.3828907012939453 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5573718547821045 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2857914865016937 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.36231231689453125 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_4_of_5‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5857747793197632 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 particle/accuracy_4_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8961988687515259 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.26546305418014526 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.985526442527771 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23119981586933136 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4360465109348297 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2361111044883728 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3856978118419647 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2292807251214981 ,jet/accuracy_3_of_3‚0.023219844326376915 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 4 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2692891061306 particle/accuracy_2_of_4‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.3568353056907654 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3210526704788208 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.3457697331905365 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3738051950931549 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.42654725909233093 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.32959771156311035 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9179894328117371 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2767210304737091 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5835675001144409 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34516283869743347 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.552322506904602 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018316958099603653 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.27669015526771545 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.019013188779354095 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2361111044883728 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.33790841698646545 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9961219429969788 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4004850685596466 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5407406687736511 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24476054310798645 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04454023018479347 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.37340065836906433 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.29274606704711914 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3701140880584717 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5320513248443604 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 particle/accuracy_7_of_7‚1 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3783804178237915 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 *jet/accuracy_1_of_4‚0.8994095325469971 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4259259104728699 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4120681583881378 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33388906717300415 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010746258310973644 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_2_of_4‚0.35473302006721497 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5310018062591553 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23341019451618195 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2669357657432556 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020086444914340973 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2689846158027649 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5424628257751465 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473652839661 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9787041544914246 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10297124832868576 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6041666865348816 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6889509558677673 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24095629155635834 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 particle/accuracy_1_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9955357313156128 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11358313262462616 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4345571994781494 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2103448063135147 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23411011695861816 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7240420579910278 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11404237896203995 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4365690052509308 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.3361111283302307 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.42074522376060486 *loss/val_total_loss‚40.973709707891786 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5648809671401978 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9949551224708557 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8104681968688965 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.38376134634017944 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9418036341667175 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22530868649482727 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3261125683784485 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.030188681557774544 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2527777850627899 particle/accuracy_2_of_6‚1 particle/accuracy_6_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28186923265457153 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4200436770915985 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5324074625968933 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9872853755950928 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.25262054800987244 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 % _timestamp‚1.7884931075574715e+09 particle/accuracy_2_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03839286044239998 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2986111044883728 particle/accuracy_6_of_6‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9883720874786377 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2576632797718048 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _step‚395 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.986160933971405 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4333333373069763 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26286065578460693 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9853035807609558 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666567325592 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9832884073257446 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.19555556774139404 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0024565381463617086 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9079862236976624 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4320371150970459 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9818211793899536 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.3256756663322449 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_6‚0.09615384787321091 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11331499367952347 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5828258395195007 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_3_of_4‚0.05783877521753311 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9822726249694824 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_3_of_5‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333730697632 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9751906991004944 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.992626428604126 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3087797462940216 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.347494512796402 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 +jet/accuracy_2_of_5‚0.47316309809684753 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4201389253139496 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2708098888397217 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 6validation_average_jet_accuracy‚0.4116474390029907 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.991669237613678 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4032319486141205 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984994530677795 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3919752836227417 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.982149064540863 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3093253970146179 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8443848490715027 ("‚ä•Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.37345197796821594 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9921483993530273 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.38177525997161865 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_1_of_7‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00869909767061472 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 epoch‚3 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.38284897804260254 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3970516324043274 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9955710172653198 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9450430274009705 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.33649423718452454 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3735632002353668 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.043152160942554474 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 jet/accuracy_4_of_7‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6499999761581421 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 6 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.268535315990448 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9980132579803467 * Purity/2FRt1SRqqt2FBt/FRt_purity‚0.375 *jet/accuracy_1_of_1‚0.4466703534126282 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 _runtime‚5207.200040473 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.42294207215309143 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.27492496371269226 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 5 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.96041339635849 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5592302680015564 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10542894154787064 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9850178956985474 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41329896450042725 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3492780923843384 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +particle/sensitivity‚0.3113715350627899 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4381847679615021 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3424730896949768 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33217164874076843 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014023544266819954 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 particle/accuracy_3_of_3‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5579860806465149 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9652777314186096 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_1_of_6‚1 particle/accuracy_3_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23119981586933136 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9591526389122009 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.369538277387619 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.39094603061676025 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5347700715065002 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8433521389961243 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3157407343387604 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.995634913444519 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9730146527290344 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897716045379639 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.30575108528137207 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_2‚1 jet/accuracy_5_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5088275074958801 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9758417010307312 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03916772082448006 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1967848390340805 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9795576333999634 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5028488039970398o!E¾]`Î̦ 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9961219429969788 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4004850685596466 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5407406687736511 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24476054310798645 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04454023018479347 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.37340065836906433 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.29274606704711914 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3701140880584717 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5320513248443604 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 particle/accuracy_7_of_7‚1 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3783804178237915 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 *jet/accuracy_1_of_4‚0.8994095325469971 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4259259104728699 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4120681583881378 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33388906717300415 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010746258310973644 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_2_of_4‚0.35473302006721497 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5310018062591553 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23341019451618195 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2669357657432556 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020086444914340973 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2689846158027649 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5424628257751465 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473652839661 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9787041544914246 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10297124832868576 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6041666865348816 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6889509558677673 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24095629155635834 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 particle/accuracy_1_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9955357313156128 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11358313262462616 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4345571994781494 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2103448063135147 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23411011695861816 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7240420579910278 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11404237896203995 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4365690052509308 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.3361111283302307 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.42074522376060486 *loss/val_total_loss‚40.973709707891786 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5648809671401978 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9949551224708557 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8104681968688965 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.38376134634017944 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9418036341667175 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22530868649482727 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3261125683784485 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.030188681557774544 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2527777850627899 particle/accuracy_2_of_6‚1 particle/accuracy_6_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28186923265457153 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4200436770915985 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5324074625968933 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9872853755950928 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.25262054800987244 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 % _timestamp‚1.7884931075574715e+09 particle/accuracy_2_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03839286044239998 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2986111044883728 particle/accuracy_6_of_6‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9883720874786377 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2576632797718048 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _step‚395 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.986160933971405 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4333333373069763 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26286065578460693 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9853035807609558 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666567325592 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9832884073257446 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.19555556774139404 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0024565381463617086 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9079862236976624 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4320371150970459 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9818211793899536 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.3256756663322449 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_6‚0.09615384787321091 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11331499367952347 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5828258395195007 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +jet/accuracy_3_of_4‚0.05783877521753311 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9822726249694824 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_3_of_5‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333730697632 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9751906991004944 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.992626428604126 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3087797462940216 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.347494512796402 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 +jet/accuracy_2_of_5‚0.47316309809684753 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4201389253139496 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2708098888397217 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 6validation_average_jet_accuracy‚0.4116474390029907 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.991669237613678 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4032319486141205 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984994530677795 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3919752836227417 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.982149064540863 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3093253970146179 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8443848490715027 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.37345197796821594 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9921483993530273 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.38177525997161865 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_1_of_7‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00869909767061472 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 epoch‚3 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.38284897804260254 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3970516324043274 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9955710172653198 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9450430274009705 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.33649423718452454 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3735632002353668 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.043152160942554474 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 jet/accuracy_4_of_7‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6499999761581421 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 6 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.268535315990448 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9980132579803467 * Purity/2FRt1SRqqt2FBt/FRt_purity‚0.375 *jet/accuracy_1_of_1‚0.4466703534126282 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 _runtime‚5207.200040473 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.42294207215309143 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.27492496371269226 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 5 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.96041339635849 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5592302680015564 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10542894154787064 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9850178956985474 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41329896450042725 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3492780923843384 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +particle/sensitivity‚0.3113715350627899 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4381847679615021 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3424730896949768 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33217164874076843 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014023544266819954 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 particle/accuracy_3_of_3‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5579860806465149 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9652777314186096 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_1_of_6‚1 particle/accuracy_3_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23119981586933136 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9591526389122009 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.369538277387619 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.39094603061676025 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5347700715065002 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8433521389961243 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3157407343387604 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.995634913444519 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9730146527290344 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897716045379639 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.30575108528137207 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_2‚1 jet/accuracy_5_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5088275074958801 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9758417010307312 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03916772082448006 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1967848390340805 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9795576333999634 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5028488039970398 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.968849241733551 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.018801964819431305 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3477678596973419 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.45671144127845764 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 jet/accuracy_7_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.38376134634017944 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04063556343317032 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2653830051422119 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2322048544883728 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ( Purity/3FRt4SRqqt*FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2581247389316559 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15227364003658295 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06984104961156845 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5744665265083313 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9101654291152954 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2625969648361206 particle/accuracy_1_of_1‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_1_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.265325665473938 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4067467451095581 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6487901210784912 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3256944715976715 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 *"Óÿ ö2Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08510638028383255 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03074757196009159 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.41130441427230835 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9892869591712952 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2648809552192688 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.36523836851119995 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.4166666865348816 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,jet/accuracy_4_of_5‚0.010618279688060284 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0219645444303751 jet/accuracy_3_of_7‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17874997854232788 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3184744119644165 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3645833432674408 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18434299528598785 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.018147870898246765 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.31998053193092346 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3523470163345337 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4945175349712372 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.4166666865348816 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2619466185569763 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.35960203409194946 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984994530677795 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.25823044776916504 +jet/accuracy_3_of_5‚0.08058755099773407 jet/accuracy_6_of_7‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5798439979553223 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2845926880836487 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28882548213005066 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 particle/accuracy_1_of_3‚1 ( Purity/3FRt4SRqqt0FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 trainer/global_step‚4835 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.535187304019928 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29611286520957947 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.43333330750465393 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333730697632 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 jet/accuracy_4_of_6‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2998655438423157 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3580247163772583 particle/accuracy_4_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5862839818000793 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2706093192100525 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.26541632413864136 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30297598242759705 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.36626681685447693 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9863973259925842 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025651821866631508 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17577949166297913 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.45621293783187866 -jet/accuracy_4_of_4‚0.0031902098562568426 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6033282279968262 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5357311964035034 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16258960962295532 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.202707439661026 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.40710073709487915 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25764936208724976 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.39430636167526245 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 'particle/f_score‚0.3828907012939453 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5573718547821045 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2857914865016937 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.36231231689453125 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_4_of_5‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5857747793197632 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 particle/accuracy_4_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8961988687515259 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.26546305418014526 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.985526442527771 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23119981586933136 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4360465109348297 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2361111044883728 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3856978118419647 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2292807251214981 ,jet/accuracy_3_of_3‚0.023219844326376915 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 4 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2692891061306 particle/accuracy_2_of_4‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.3568353056907654 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3210526704788208 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.3457697331905365 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3738051950931549 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.42654725909233093 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.32959771156311035 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9179894328117371 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2767210304737091 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5835675001144409 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34516283869743347 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.552322506904602 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018316958099603653 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.27669015526771545 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.019013188779354095 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2361111044883728 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.33790841698646545 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1‹¡3œÏ‚(:‘ ÀúèÔ³‰É­ gpu.0.gpu‚2.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚33.0 gpu.0.powerWatts‚114.067' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.813399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;³ƒÐ‚(:ø ÀúèÔêÀó² network.recv‚ 1.73384e+06# disk.nvme0c0n1.in‚ 20910.5859375& disk.nvme0n1.out‚130181.0947265625 disk.dm-0.in‚ 20910.8671875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7578125 disk.sda.in‚0 disk.sda1.in‚0) disk./.usagePercent‚78.83684354138458% disk./.usageGB‚1174.7259407043457( disk.nvme0c0n1.out‚130181.0947265625 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1678.18359375 proc.cpu.threads‚17 disk.dm-1.in‚0 network.sent‚ 1.1539849e+07! disk.nvme0n1.in‚ 20910.8671875# disk.dm-0.out‚130181.0947265625 disk.sda3.out‚ 155.02734375 disk.sda.out‚ 155.02734375$ memory_percent‚89.75718021392822* proc.memory.percent‚17.921781539916992 disk.dm-1.out‚ 155.02734375 disk.sda2.out‚0" proc.memory.rssMB‚ 2936.3046875 cpu‚25.268360491567844ª·ÿ;2Ñ- _wandbruntime‚5226  _runtime‚5226^¢ÇÒÁ & lr-AdamW/pg1‚0.0005928925831346417 & lr-AdamW/pg2‚0.0005928925831346417 trainer/global_step‚4849 % _timestamp‚1.7884931263656888e+09 _runtime‚5226.006386082 _step‚396ˆ¶Ê;ÌÓÆ % _timestamp‚1.7884931263656888e+09 _runtime‚5226.006386082 _step‚396 & lr-AdamW/pg1‚0.0005928925831346417 & lr-AdamW/pg2‚0.0005928925831346417 trainer/global_step‚4849Œ6.Tä2Ô- _wandbruntime‚5226  _runtime‚5226Y”ÓÃö'ÕðO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85116916 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4991333459456 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5207167682560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204627968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4849 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192853663 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27123783 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚339823781208576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48810177 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4973365001918 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4991333459456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48810175 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75933958 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85116916 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107736747 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚331099290384624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344815096665088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107736747 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚331099307423984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚339823763205632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4992106104320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107713731 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚344815096665088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4991333459456 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5207167682560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204627968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4849 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192853663 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27123783 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚339823781208576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48810177 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4973365001918 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4991333459456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48810175 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75933958 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85116916 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107736747 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚331099290384624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344815096665088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107736747 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚331099307423984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚339823763205632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4992106104320 =7DeviceStatsMonitor.on_train_batcÞÇ‘è< h_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107713731 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚344815096665088 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75948725 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5208004274176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75949240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27128902 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚377371812111360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚377371788171264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339891463144960 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107758484 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4993110863360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107735442 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192892274 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48819825 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4975136790416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 trainer/global_step‚4849 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192892274 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339891481147904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚382580016805888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 _runtime‚5226.869137844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85133788 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344883801350144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚331165373859744 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192869230 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚336139722962180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344884592011264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85133790 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339891463144960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48819823 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚336140510650160 % _timestamp‚1.7884931272282822e+09 _step‚398 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4992338205184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4993110863360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27129415 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5208204694528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4974366141796 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 121550929922Ó5‰'ÙÿM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192892274 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48819825 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4975136790416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 trainer/global_step‚4849 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192892274 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339891481147904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚382580016805888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 _runtime‚5226.869137844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85133788 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344883801350144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚331165373859744 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192869230 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚336139722962180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344884592011264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85133790 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339891463144960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48819823 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚336140510650160 % _timestamp‚1.7884931272282822e+09 _step‚398 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4992338205184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4993110863360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27129415 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5208204694528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4974366141796 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107735442 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75948725 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5208004274176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75949240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27128902 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚377371812111360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚377371788171264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339891463144960 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107758484 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4993110863360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107735442 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489ŽÀ­Ð2Ú- _wandbruntime‚5226  _runtime‚5226º’¯‰]Û× 1loss/FBt1/assignment_loss‚0.20677675134275594 1loss/FBt2/assignment_loss‚0.21555273613521422 1loss/FBt4/assignment_loss‚0.22629399552350632 /loss/FRt1/detection_loss‚2.4220407707868397 trainer/global_step‚4849 0loss/FBt3/assignment_loss‚0.2677002812311652 %loss/total_loss‚39.31154343280537 _runtime‚5226.870662627 0loss/FRt1/assignment_loss‚2.9027310706473686 0loss/FRt2/assignment_loss‚2.9962287385289263 2loss/SRqqt1/assignment_loss‚1.2778104282915592 1loss/SRqqt2/detection_loss‚2.4937345773884743 1loss/SRqqt3/detection_loss‚1.2475110013515522 epoch‚4 1loss/SRqqt4/detection_loss‚1.1334627845981082 /loss/FBt4/detection_loss‚0.6402037811849047 % _timestamp‚1.7884931272305114e+09 1loss/SRqqt3/assignment_loss‚1.076308475043147 /loss/FRt3/detection_loss‚1.8782600227155184 /loss/FRt4/detection_loss‚1.6903306634298392 0loss/SRqqt1/detection_loss‚2.684763550758362 .loss/FBt2/detection_loss‚2.625614786148071 /loss/FRt4/assignment_loss‚1.494590733732496 /loss/FBt1/detection_loss‚2.8761611779530845 /loss/FBt3/detection_loss‚2.3093165372099196 _step‚399 0loss/FRt3/assignment_loss‚1.9370788356714082 2loss/SRqqt2/assignment_loss‚1.3570162387508335 2loss/SRqqt4/assignment_loss‚0.8931141612043635 .loss/FRt2/detection_loss‚2.458941333177958 þ übÜÜ trainer/global_step‚4849 0loss/FBt3/assignment_loss‚0.2677002812311652 %loss/total_loss‚39.31154343280537 _runtime‚5226.870662627 0loss/FRt1/assignment_loss‚2.9027310706473686 0loss/FRt2/assignment_loss‚2.9962287385289263 2loss/SRqqt1/assignment_loss‚1.2778104282915592 1loss/SRqqt2/detection_loss‚2.4937345773884743 1loss/SRqqt3/detection_loss‚1.2475110013515522 epoch‚4 1loss/SRqqt4/detection_loss‚1.1334627845981082 /loss/FBt4/detection_loss‚0.6402037811849047 % _timestamp‚1.7884931272305114e+09 1loss/SRqqt3/assignment_loss‚1.076308475043147 /loss/FRt3/detection_loss‚1.8782600227155184 /loss/FRt4/detection_loss‚1.6903306634298392 0loss/SRqqt1/detection_loss‚2.684763550758362 .loss/FBt2/detection_loss‚2.625614786148071 /loss/FRt4/assignment_loss‚1.494590733732496 /loss/FBt1/detection_loss‚2.8761611779530845 /loss/FBt3/detection_loss‚2.3093165372099196 _step‚399 0loss/FRt3/assignment_loss‚1.9370788356714082 2loss/SRqqt2/assignment_loss‚1.3570162387508335 2loss/SRqqt4/assignment_loss‚0.8931141612043635 .loss/FRt2/detection_loss‚2.458941333177958 1loss/FBt1/assignment_loss‚0.20677675134275594 1loss/FBt2/assignment_loss‚0.21555273613521422 1loss/FBt4/assignment_loss‚0.22629399552350632 /loss/FRt1/detection_loss‚2.4220407707868397ôCÖÝ‚(:„ Ïúè԰ৰ gpu.0.gpu‚8.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.52' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»'Ù§vÞ‚(:ë ÏúèÔ†‰Ó± disk.sda3.out‚155.5 disk.drbd1260.out‚ 0.76171875" disk.nvme0n1.in‚20993.91796875 disk.dm-0.in‚20993.91796875) proc.memory.availableMB‚ 1678.03515625 network.recv‚ 1.739365e+06) disk./.usagePercent‚78.83671579362499% disk./.usageGB‚1174.7240371704102 disk.dm-1.out‚155.5 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2936.3046875 disk.sda3.in‚0 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚20993.91796875) disk.nvme0c0n1.out‚130740.43212890625' disk.nvme0n1.out‚130740.43212890625$ disk.dm-0.out‚130740.43212890625 cpu‚25.267850733831896 proc.cpu.threads‚17 network.sent‚ 1.1663536e+07 disk.sda.in‚0 disk.sda.out‚155.5 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚89.75808620452881* proc.memory.percent‚17.921781539916992'` ÷’ß‚(:‡ ÞúèÔ“¼ù¬ gpu.0.gpu‚21.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.328' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2656 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù6xŠà‚(:ÿ ÞúèÔ®™î® network.sent‚ 1.1772643e+07$ disk./.usageGB‚1174.126449584961 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0n1.out‚131102.19580078125 disk.dm-0.in‚20993.91796875 disk.sda3.in‚0 cpu‚25.267359035052543 disk.sda2.out‚0) disk.nvme0c0n1.out‚131102.19580078125 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚89.75751399993896) proc.memory.availableMB‚ 1678.12890625 proc.cpu.threads‚17 network.recv‚ 1.744876e+06) disk./.usagePercent‚78.79661118935611$ disk.nvme0c0n1.in‚20993.91796875$ disk.dm-0.out‚131102.19580078125 disk.dm-1.out‚ 155.91796875 disk.drbd1260.out‚ 0.76171875 disk.sda.in‚0 disk.sda.out‚ 155.91796875 disk.sda2.in‚0" disk.nvme0n1.in‚20993.91796875 disk.sda3.out‚ 155.91796875" proc.memory.rssMB‚ 2936.3046875* proc.memory.percent‚17.921781539916992|†oI’á‚(:‡ íúèÔý—±® gpu.0.gpu‚45.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚150.032' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0064 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢¨½Ð…â‚(:ú íúèÔ…³ø® proc.cpu.threads‚17) disk./.usagePercent‚78.79818230759375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.in‚20993.91796875$ disk.dm-0.out‚131536.09423828125" proc.memory.rssMB‚ 2936.3046875 disk.sda3.out‚ 156.44140625 disk.drbd1260.out‚0.765625) disk.nvme0c0n1.out‚131536.09423828125' proc.memory.availableMB‚ 1677.640625* proc.memory.percent‚17.921781539916992 cpu‚25.266935518077684 network.sent‚ 1.1777031e+07# disk./.usageGB‚1174.14986038208 disk.drbd1260.in‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚20993.91796875 disk.dm-0.in‚20993.91796875 network.recv‚ 1.749551e+06 disk.sda3.in‚0 disk.dm-1.out‚ 156.44140625 disk.sda.out‚ 156.44140625 disk.sda2.in‚0' disk.nvme0n1.out‚131536.09423828125$ memory_percent‚89.76049423217773»õ»n2ã- _wandbruntime‚5277  _runtime‚5277þ„¶ÇäÁ & lr-AdamW/pg1‚0.0005879092246101835 & lr-AdamW/pg2‚0.0005879092246101835 trainer/global_step‚4899 % _timestamp‚1.7884931773919702e+09 _runtime‚5277.032349214 _step‚400‚gÚýÌåÆ _runtime‚5277.032349214 _step‚400 & lr-AdamW/pg1‚0.0005879092246101835 & lr-AdamW/pg2‚0.0005879092246101835 trainer/global_step‚4899 % _timestamp‚1.7884931773919702e+09'l±2æ- _wandbruntime‚5277  _runtime‚5277¶@/›ï çðO BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚348250345331712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85960616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27385153 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5042357804544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85960614 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343208760172032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76677728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5259020999680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5024206235438 _runtime‚5277.033483381 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 trainer/global_step‚4899 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 O?DeviceStatsMonitor.on_train_batch_start/reserved_â›×¨bytes.all.peak‚ 12155092992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343208778174976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚348250345331712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 % _timestamp‚1.7884931773930018e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49292575 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386323106535424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5259225627648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343208760172032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚348251135979520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚381063856967680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108802633 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚348251135979520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚339426047759650 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334402612172624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85960616 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 _step‚401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76678261 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194786265 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27385684 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108802633 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0R½Ògû'èõO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5259225627648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343208760172032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚348251135979520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚381063856967680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108802633 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚348251135979520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚339426047759650 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334402612172624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85960616 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 _step‚401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76678261 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194786265 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27385684 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108802633 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚348250345331712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85960616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27385153 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5042357804544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85960614 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343208760172032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76677728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5259020999680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5024206235438 _runtime‚5277.033483381 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 trainer/global_step‚4899 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343208778174976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚348250345331712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 % _timestamp‚1.7884931773930018e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49292575 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386323106535424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489‘ÂÀ 2é- _wandbruntime‚5277  _runtime‚5277C }I'êúM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194824876 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5043362563584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76693133 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 trainer/global_step‚4899 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108824344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚348319840677888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334468678608384 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884931782512171e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76692586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5260259120128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27390363 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49302225 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194824876 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85977490 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚348319840677888 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108824344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚381139229250560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚339493115335288 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334468695647744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚348319050016768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5042589905408 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27390908 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85977488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85977488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85977490ÉF¿}©ëÿM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27390908 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85977488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85977488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85977490 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194824876 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5043362563584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76693133 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 trainer/global_step‚4899 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108824344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚348319840677888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334468678608384 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884931782512171e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76692586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5260259120128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27390363 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49302225 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194824876 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85977490 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚348319840677888 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108824344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚381139229250560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚339493115335288 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334468695647744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚348319050016768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5042589905408 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39’^Y6Õ2ì- _wandbruntime‚5277  _runtime‚5277 B)L[íÕ _runtime‚5277.893136153 1loss/SRqqt3/assignment_loss‚1.123385796439834 1loss/SRqqt1/detection_loss‚2.1590674193792565 2loss/SRqqt4/assignment_loss‚1.0227416805478173 1loss/FBt3/assignment_loss‚0.17394111741768112 /loss/FRt4/detection_loss‚1.7679664289578796 %loss/total_loss‚39.33253716898964 0loss/FRt1/assignment_loss‚3.0412356229055497 1loss/SRqqt2/assignment_loss‚2.067685153069241 /loss/FRt1/detection_loss‚2.4364624995560873 /loss/FRt2/detection_loss‚2.7590467001710675 1loss/SRqqt3/detection_loss‚1.4366853314859327 .loss/FBt1/detection_loss‚2.455384660512209 -loss/FBt2/detection_loss‚2.36788672208786 _step‚403 /loss/FRt3/assignment_loss‚2.362601914670732 0loss/FRt4/assignment_loss‚1.1205107927322389 /loss/FBt3/detection_loss‚1.9945375702597878 .loss/FRt2/assignment_loss‚2.62760831287929 1loss/FBt2/assignment_loss‚0.20447706210608077 1loss/SRqqt2/detection_loss‚2.2923605931656703 /loss/FBt4/detection_loss‚0.7645304105834022 % _timestamp‚1.7884931782529535e+09 2loss/SRqqt1/assignment_loss‚1.5777978949075522 1loss/FBt4/assignment_loss‚0.08568924707342032 1loss/FBt1/assignment_loss‚0.41293236846104264 /loss/FRt3/detection_loss‚2.1591128669679165 1loss/SRqqt4/detection_loss‚0.9188890026520882 epoch‚4 trainer/global_step‚4899¶³¶`îÚ 1loss/FBt1/assignment_loss‚0.41293236846104264 /loss/FRt3/detection_loss‚2.1591128669679165 1loss/SRqqt4/detection_loss‚0.9188890026520882 epoch‚4 trainer/global_step‚4899 _runtime‚5277.893136153 1loss/SRqqt3/assignment_loss‚1.123385796439834 1loss/SRqqt1/detection_loss‚2.1590674193792565 2loss/SRqqt4/assignment_loss‚1.0227416805478173 1loss/FBt3/assignment_loss‚0.17394111741768112 /loss/FRt4/detection_loss‚1.7679664289578796 %loss/total_loss‚39.33253716898964 0loss/FRt1/assignment_loss‚3.0412356229055497 1loss/SRqqt2/assignment_loss‚2.067685153069241 /loss/FRt1/detection_loss‚2.4364624995560873 /loss/FRt2/detection_loss‚2.7590467001710675 1loss/SRqqt3/detection_loss‚1.4366853314859327 .loss/FBt1/detection_loss‚2.455384660512209 -loss/FBt2/detection_loss‚2.36788672208786 _step‚403 /loss/FRt3/assignment_loss‚2.362601914670732 0loss/FRt4/assignment_loss‚1.1205107927322389 /loss/FBt3/detection_loss‚1.9945375702597878 .loss/FRt2/assignment_loss‚2.62760831287929 1loss/FBt2/assignment_loss‚0.20447706210608077 1loss/SRqqt2/detection_loss‚2.2923605931656703 /loss/FBt4/detection_loss‚0.7645304105834022 % _timestamp‚1.7884931782529535e+09 2loss/SRqqt1/assignment_loss‚1.5777978949075522 1loss/FBt4/assignment_loss‚0.08568924707342032“2™0’ï‚(:‡ üúèÔÀ½‘® gpu.0.gpu‚26.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.126' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ«^‚ð‚(:÷ üúèԥБ¯ proc.cpu.threads‚17% disk./.usageGB‚1174.1498641967773" disk.nvme0n1.in‚20993.91796875 disk.dm-0.in‚20993.91796875( proc.memory.availableMB‚ 1676.8046875" proc.memory.rssMB‚ 2936.3046875$ disk.nvme0c0n1.in‚20993.91796875) disk.nvme0c0n1.out‚131728.64111328125$ disk.dm-0.out‚131728.64111328125 disk.sda3.out‚ 156.859375 disk.drbd1260.in‚0 disk.drbd1260.out‚0.765625 disk.sda.in‚0 network.recv‚ 1.754042e+06) disk./.usagePercent‚78.79818256360129 disk.sda2.in‚0' disk.nvme0n1.out‚131728.64111328125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 156.859375$ memory_percent‚89.76559638977051 network.sent‚ 1.1782902e+07 disk.sda2.out‚0 disk.dm-1.out‚ 156.859375 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.921781539916992 cpu‚25.266512893459204\S0œñ‚(:‘ ‹ûèÔ“Ù‹¯ gpu.0.gpu‚2.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.594' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.918799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ä9©…ò‚(:ú ‹ûèÔÕüÜ´ disk.sda1.out‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚21171.21484375" disk.nvme0n1.in‚21171.21484375& disk.nvme0n1.out‚131945.5009765625 disk.dm-0.in‚21171.21484375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.76953125# disk.dm-0.out‚131945.5009765625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 157.2734375 disk.sda.in‚0$ memory_percent‚89.76695537567139) proc.memory.availableMB‚ 1676.58203125 network.recv‚ 1.760159e+06) disk./.usagePercent‚78.79660530118284 disk.sda2.out‚0( disk.nvme0c0n1.out‚131945.5009765625 disk.sda.out‚ 157.2734375* proc.memory.percent‚17.921781539916992 cpu‚25.266041608136437 proc.cpu.threads‚17 network.sent‚ 1.1892934e+07% disk./.usageGB‚1174.1263618469238 disk.sda1.in‚0 disk.sda3.out‚ 157.2734375" proc.memory.rssMB‚ 2936.3046875´³Ê’ó‚(:‡ šûèÔþÇ¥¯ gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nΨ‚ô‚(:÷ šûèÔÚ¢ó¯ disk.sda.out‚ 157.8671875 network.sent‚ 1.1898086e+07) disk./.usagePercent‚78.79190781893706% disk./.usageGB‚1174.0563659667969# disk.dm-0.out‚132163.1376953125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.76953125 disk.sda1.out‚0 disk.dm-0.in‚21171.21484375 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚21171.21484375( disk.nvme0c0n1.out‚132163.1376953125$ memory_percent‚89.76850509643555 cpu‚25.26562534442873 proc.cpu.threads‚17 disk.sda3.out‚ 157.8671875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚21171.21484375& disk.nvme0n1.out‚132163.1376953125" proc.memory.rssMB‚ 2936.3046875 network.recv‚ 1.763314e+06 disk.dm-1.out‚ 157.8671875 disk.drbd1260.in‚0' proc.memory.availableMB‚ 1676.328125* proc.memory.percent‚17.921781539916992ȲKU’õ‚(:‡ ©ûèÔº¶¦­ gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.291' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—nF¨€ö‚(:õ ©ûèÔ¿Ž¯ network.recv‚ 1.767843e+06( disk./.usagePercent‚78.8009886621841 disk.dm-1.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚21171.21484375' disk.nvme0c0n1.out‚132773.630859375 disk.dm-0.in‚21171.21484375* proc.memory.percent‚17.921781539916992 disk.drbd1260.out‚ 0.7734375 disk.sda.in‚0 disk.sda.out‚ 158.3046875 disk.sda1.in‚0% disk.nvme0n1.out‚132773.630859375$ memory_percent‚89.76705074310303) proc.memory.availableMB‚ 1676.56640625" proc.memory.rssMB‚ 2936.3046875 network.sent‚ 1.1902509e+07% disk./.usageGB‚1174.1916770935059 disk.sda3.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚21171.21484375" disk.dm-0.out‚132773.630859375 cpu‚25.265080667426837 proc.cpu.threads‚17 disk.sda3.out‚ 158.3046875 disk.dm-1.out‚ 158.3046875 disk.sda1.out‚0 disk.sda2.in‚0é)R2÷- _wandbruntime‚5326  _runtime‚5326ÕL‚èÇøÁ & lr-AdamW/pg1‚0.0005828858977184552 & lr-AdamW/pg2‚0.0005828858977184552 trainer/global_step‚4949 % _timestamp‚1.7884932265261526e+09 _runtime‚5326.166452508 _step‚404–D[ ÌùÆ & lr-AdamW/pg1‚0.0005828858977184552 & lr-AdamW/pg2‚0.0005828858977184552 trainer/global_step‚4949 % _timestamp‚1.7884932265261526e+09 _runtime‚5326.166452508 _step‚404”Œe,2ú- _wandbruntime‚5326  _runtime‚5326°Ô¤'ö'ûðO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196718867 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390142415144448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚384831274106880 % _timestamp‚1.7884932265270786e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384831298046976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚337705950999984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86804314 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27647968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390142650003968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5092609504768 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109891535 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚342780227820530 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5091836859904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚346593775141376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5091836859904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234859520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5311141037568 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196695849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚346593757138432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5311351956992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚337705933960624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109891535 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351685593998336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚405 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210919424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _runtime‚5326.167498178 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86804314 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624)£Rñ"üõO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384831298046976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚337705950999984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86804314 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27647968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390142650003968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5092609504768 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109891535 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚342780227820530 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5091836859904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚346593775141376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5091836859904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234859520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5311141037568 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196695849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚346593757138432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5311351956992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚337705933960624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109891535 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351685593998336 RBDeviceStatsAÌwÔ Monitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚405 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210919424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _runtime‚5326.167498178 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86804314 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196718867 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390142415144448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚384831274106880 % _timestamp‚1.7884932265270786e+09•§×o$2ý- _wandbruntime‚5327  _runtime‚5327e–‹w'þúM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 _step‚406 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196757478 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5075277960632 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196757478 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196734434 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚351755089344512 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚342846507708396 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86821190 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27652760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚384906646389760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5312183307776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109936288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5312385825280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337772000396384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5093614263808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4949 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109913246 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109936288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196734434 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86821188 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49784625 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77437383 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86821190 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5093614263808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚346661475080704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115}üd·'ÿÿM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 _step‚406 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196757478 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5075277960632 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196757478 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196734434 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚351755089344512 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚342846507708396 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86821190 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27652760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚384906646389760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5312183307776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109936288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5312385825280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337772000396384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5093614263808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4949 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109913246 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109936288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196734434 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86821188 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49784625 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77437383 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86821190 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5093614263808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚346661475080704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489–ÖO2€- _wandbruntime‚5327  _runtime‚5327áuŠËWÑ 2loss/SRqqt2/assignment_loss‚1.7107897377931154 0loss/FBt3/assignment_loss‚0.3098967234497347 .loss/FRt4/detection_loss‚1.762007139461196 _step‚407 .loss/FRt2/detection_loss‚2.553095692768718 .loss/FBt2/detection_loss‚3.516652965545654 /loss/FRt1/assignment_loss‚2.872261335070317 /loss/FRt4/assignment_loss‚1.357643829739612 2loss/SRqqt1/assignment_loss‚1.5845113049239614 2loss/SRqqt4/assignment_loss‚0.9835668958225091 1loss/FBt2/assignment_loss‚0.08634421497117728 .loss/FBt1/detection_loss‚3.542346199353536 /loss/FBt4/detection_loss‚0.7013145708593965 epoch‚4 trainer/global_step‚4949 /loss/FRt1/detection_loss‚2.3092383309625664 1loss/SRqqt3/detection_loss‚1.3335696362663765 1loss/SRqqt4/detection_loss‚1.0168548195924558 %loss/total_loss‚42.17952452954161 % _timestamp‚1.7884932273765101e+09 /loss/FRt3/detection_loss‚2.3449526131153107 .loss/FBt3/detection_loss‚2.287624885638555 0loss/FRt2/assignment_loss‚3.6112716421484947 2loss/SRqqt3/assignment_loss‚1.3000853526330478 0loss/SRqqt1/detection_loss‚2.048206379247266 1loss/SRqqt2/detection_loss‚2.0056759929045653 _runtime‚5327.016677553 0loss/FRt3/assignment_loss‚2.3082225259981657 .loss/FBt1/assignment_loss‚0.53165486206611 1loss/FBt4/assignment_loss‚0.10173687920976862¾…ª\‚Ö /loss/FRt1/assignment_loss‚2.872261335070317 /loss/FRt4/assignment_loss‚1.357643829739612 2loss/SRqqt1/assignment_loss‚1.5845113049239614 2loss/SRqqt4/assignment_loss‚0.9835668958225091 1loss/FBt2/assignment_loss‚0.08634421497117728 .loss/FBt1/detection_loss‚3.542346199353536 /loss/FBt4/detection_loss‚0.7013145708593965 epoch‚4 trainer/global_step‚4949 /loss/FRt1/detection_loss‚2.3092383309625664 1loss/SRqqt3/detection_loss‚1.3335696362663765 1loss/SRqqt4/detection_loss‚1.0168548195924558 %loss/total_loss‚42.17952452954161 % _timestamp‚1.7884932273765101e+09 /loss/FRt3/detection_loss‚2.3449526131153107 .loss/FBt3/detection_loss‚2.287624885638555 0loss/FRt2/assignment_loss‚3.6112716421484947 2loss/SRqqt3/assignment_loss‚1.3000853526330478 0loss/SRqqt1/detection_loss‚2.048206379247266 1loss/SRqqt2/detection_loss‚2.0056759929045653 _runtime‚5327.016677553 0loss/FRt3/assignment_loss‚2.3082225259981657 .loss/FBt1/assignment_loss‚0.53165486206611 1loss/FBt4/assignment_loss‚0.10173687920976862 2loss/SRqqt2/assignment_loss‚1.7107897377931154 0loss/FBt3/assignment_loss‚0.3098967234497347 .loss/FRt4/detection_loss‚1.762007139461196 _step‚407 .loss/FRt2/detection_loss‚2.553095692768718 .loss/FBt2/detection_loss‚3.516652965545654—H¾š}’ƒ‚(:‡ ¸ûèÔñê° gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.346' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6692 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šÃg‚„‚(:÷ ¸ûèÔ¹À² network.sent‚ 1.190837e+07% disk./.usageGB‚1173.1555557250977 disk.sda3.in‚0" proc.memory.rssMB‚ 2936.3046875 cpu‚25.264591013624287 proc.cpu.threads‚17) disk./.usagePercent‚78.73145368778648$ disk.nvme0c0n1.in‚21171.21484375" disk.nvme0n1.in‚21171.21484375 disk.dm-0.in‚21171.21484375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7734375 disk.sda.in‚0 network.recv‚ 1.772144e+06' disk.nvme0c0n1.out‚133215.576171875% disk.nvme0n1.out‚133215.576171875 disk.sda.out‚ 158.74609375 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚89.76788520812988( proc.memory.availableMB‚ 1676.4296875" disk.dm-0.out‚133215.576171875 disk.sda3.out‚ 158.74609375 disk.dm-1.out‚ 158.74609375 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.921781539916992×nzœ…‚(:‘ ÇûèÔžž€¯ gpu.0.gpu‚2.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.614' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.922800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[IYV†‚(:ô ÇûèÔ¹ü±² disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.77734375 disk.sda.in‚0 disk.sda.out‚ 159.140625# disk.nvme0c0n1.in‚ 21347.6953125 disk.sda3.in‚0 network.recv‚ 1.778588e+06 disk.sda2.out‚0 disk.sda3.out‚ 159.140625" proc.memory.rssMB‚ 2936.3046875* proc.memory.percent‚17.921781539916992 proc.cpu.threads‚17 network.sent‚ 1.2016774e+07 disk.sda1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 21347.6953125' disk.nvme0n1.out‚133551.27490234375 disk.dm-0.in‚ 21347.6953125' proc.memory.availableMB‚ 1676.671875 cpu‚25.264091997038367 disk.sda1.out‚0) disk.nvme0c0n1.out‚133551.27490234375$ disk.dm-0.out‚133551.27490234375 disk.dm-1.in‚0 disk.dm-1.out‚ 159.140625$ memory_percent‚89.76640701293945) disk./.usagePercent‚78.73327569340766$ disk./.usageGB‚1173.182704925537x™¥;’‡‚(:‡ ÖûèÔ‡‡°­ gpu.0.gpu‚36.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.732' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3464 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PPiòyˆ‚(:î ÖûèÔ˜Š“° disk.drbd1260.out‚ 0.77734375# disk.nvme0c0n1.in‚ 21347.6953125) disk.nvme0c0n1.out‚133646.58349609375) disk./.usagePercent‚78.73327569340766! disk.nvme0n1.in‚ 21347.6953125 disk.sda3.in‚0 disk.sda3.out‚159.5625 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1676.1796875 cpu‚25.263570139304633 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚159.5625 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2936.3046875 network.sent‚ 1.2021812e+07 network.recv‚ 1.78206e+06$ disk.dm-0.out‚133646.58349609375$ memory_percent‚89.76941108703613* proc.memory.percent‚17.921781539916992 proc.cpu.threads‚17$ disk./.usageGB‚1173.182704925537' disk.nvme0n1.out‚133646.58349609375 disk.dm-0.in‚ 21347.6953125 disk.dm-1.out‚159.5625 disk.drbd1260.in‚0õÃDŸ2‰- _wandbruntime‚5372  _runtime‚5372ôàØ‚ÆŠÀ & lr-AdamW/pg1‚0.0005778236493132521 & lr-AdamW/pg2‚0.0005778236493132521 trainer/global_step‚4999 $ _timestamp‚1.788493272547146e+09 _runtime‚5372.187498679 _step‚408Ž‚†±Ë‹Å & lr-AdamW/pg1‚0.0005778236493132521 & lr-AdamW/pg2‚0.0005778236493132521 trainer/global_step‚4999 $ _timestamp‚1.788493272547146e+09 _runtime‚5372.187498679 _step‚408˜iZï@2Œ- _wandbruntime‚5372  _runtime‚5372•c-ðO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388598691246080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5123576757242 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111003453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198651469 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 % _timestamp‚1.7884932725481749e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5142088560128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚349978754104832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393961894253568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5363203007488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341009272787984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5124347405654 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _runtime‚5372.188599283 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388598715186176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5142861204992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5142088560128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚ÐÛ[É23016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87648016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355120842664960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚355121633312768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27910588 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50257377 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚4999 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚349978772107776 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27911115 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110980437 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78167963 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚349978754104832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393961894253568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5363203007488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341009272787984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5124347405654 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _runtime‚5372.188599283 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388598715186176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5142861204992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5142088560128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87648016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355120842664960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚355121633312768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27910588 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50257377 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚4999 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚349978772107776 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27911115 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110980437 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78167963 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87664888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50267023 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78182856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27915833 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394038502059008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341075339223744 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚355190338011136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87664890 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27916374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111025190 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884932734107509e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5143865964032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111025190 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚346199900081504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388674063528960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5125348545740 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87664888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50267025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _runtime‚5373.051786263 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚355190338011136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198690080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4999 _step‚410 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111002148 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341075322184384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚350046454044160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232749056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚394038269309952 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352uBMÛç‘ÿM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341075339223744 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚355190338011136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87664890 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27916374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111025190 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884932734107509e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5143865964032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111025190 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚346199900081504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388674063528960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5125348545740 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87664888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50267025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _runtime‚5373.051786263 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚355190338011136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198690080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4999 _step‚410 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111002148 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341075322184384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚350046454044160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232749056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 XDDeviceStatsMon8»®nitor.on_train_batch_end/inactive_split_bytes.all.freed‚394038269309952 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78183399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚355189547350016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚350046472047104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5364438530048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚346200687769484 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87664888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50267023 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78182856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27915833 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394038502059008š2åþ2’- _wandbruntime‚5373  _runtime‚5373S,§é\“Ö 0loss/FRt1/assignment_loss‚3.5166554799894008 1loss/SRqqt3/assignment_loss‚1.237747812377555 0loss/SRqqt4/assignment_loss‚0.93261056331734 1loss/FBt3/assignment_loss‚0.06620840792840516 trainer/global_step‚4999 $ _timestamp‚1.788493273412978e+09 _step‚411 2loss/SRqqt2/assignment_loss‚1.6206645956810783 1loss/FBt2/assignment_loss‚0.07546244669686075 .loss/FRt1/detection_loss‚2.494810715317726 /loss/FRt2/detection_loss‚2.6261211861583624 0loss/FRt2/assignment_loss‚2.7224223999416126 /loss/FRt3/assignment_loss‚2.206594344515067 /loss/FRt3/detection_loss‚1.9812003983041415 1loss/SRqqt4/detection_loss‚1.1126398070597312 _runtime‚5373.053163098 2loss/SRqqt1/assignment_loss‚1.3810549470492535 /loss/FBt1/detection_loss‚2.7835052087903023 .loss/FBt3/detection_loss‚2.382480106570504 &loss/total_loss‚39.446097740438205 epoch‚4 0loss/FRt4/assignment_loss‚1.3253002516601398 1loss/FBt4/assignment_loss‚0.11277390431643974 /loss/FBt4/detection_loss‚0.6742580796838464 1loss/FBt1/assignment_loss‚0.08876262733237184 /loss/FBt2/detection_loss‚2.8819845840334892 /loss/FRt4/detection_loss‚1.4052912059199552 0loss/SRqqt1/detection_loss‚2.262812367743916 1loss/SRqqt2/detection_loss‚2.3901814380112816 1loss/SRqqt3/detection_loss‚1.1645548620394297õšIa”Û .loss/FBt3/detection_loss‚2.382480106570504 &loss/total_loss‚39.446097740438205 epoch‚4 0loss/FRt4/assignment_loss‚1.3253002516601398 1loss/FBt4/assignment_loss‚0.11277390431643974 /loss/FBt4/detection_loss‚0.6742580796838464 1loss/FBt1/assignment_loss‚0.08876262733237184 /loss/FBt2/detection_loss‚2.8819845840334892 /loss/FRt4/detection_loss‚1.4052912059199552 0loss/SRqqt1/detection_loss‚2.262812367743916 1loss/SRqqt2/detection_loss‚2.3901814380112816 1loss/SRqqt3/detection_loss‚1.1645548620394297 0loss/FRt1/assignment_loss‚3.5166554799894008 1loss/SRqqt3/assignment_loss‚1.237747812377555 0loss/SRqqt4/assignment_loss‚0.93261056331734 1loss/FBt3/assignment_loss‚0.06620840792840516 trainer/global_step‚4999 $ _timestamp‚1.788493273412978e+09 _step‚411 2loss/SRqqt2/assignment_loss‚1.6206645956810783 1loss/FBt2/assignment_loss‚0.07546244669686075 .loss/FRt1/detection_loss‚2.494810715317726 /loss/FRt2/detection_loss‚2.6261211861583624 0loss/FRt2/assignment_loss‚2.7224223999416126 /loss/FRt3/assignment_loss‚2.206594344515067 /loss/FRt3/detection_loss‚1.9812003983041415 1loss/SRqqt4/detection_loss‚1.1126398070597312 _runtime‚5373.053163098 2loss/SRqqt1/assignment_loss‚1.3810549470492535 /loss/FBt1/detection_loss‚2.7835052087903023›ršé”•‚(:ô åûèÔÃøد network.recv‚ 1.786817e+06 disk.sda1.out‚0 disk.dm-0.in‚ 21347.6953125 disk.sda3.out‚ 159.94921875 disk.drbd1260.out‚0.78125 disk.sda.in‚0* proc.memory.percent‚17.921781539916992 cpu‚25.262988325626043) disk./.usagePercent‚78.74013618331024% disk./.usageGB‚1173.2849311828613 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 21347.6953125' disk.nvme0c0n1.out‚133916.591796875! disk.nvme0n1.in‚ 21347.6953125 disk.sda3.in‚0$ memory_percent‚89.76986408233643 network.sent‚ 1.2027951e+07 disk.sda1.in‚0% disk.nvme0n1.out‚133916.591796875" disk.dm-0.out‚133916.591796875 disk.drbd1260.in‚0 disk.sda.out‚ 159.94921875) proc.memory.availableMB‚ 1676.10546875" proc.memory.rssMB‚ 2936.3046875 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 159.94921875 proc.cpu.threads‚17$–‚(:’ åûèÔÎ×­¯ gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.398' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.879599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡æ#µ—‚(:’ ôûèÔ¦¦ž­ gpu.0.gpu‚26.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.886' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.977199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wfI]˜‚(:ö ôûèÔ쉯 proc.cpu.threads‚17! disk.nvme0n1.in‚ 21347.6953125' disk.nvme0n1.out‚134288.53955078125 disk.dm-0.in‚ 21347.6953125 disk.sda3.in‚0 disk.sda3.out‚ 160.30078125 disk.dm-1.out‚ 160.30078125 disk.drbd1260.out‚0.78125$ memory_percent‚89.76836204528809) disk./.usagePercent‚78.74851018975524 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.262502492217592 network.sent‚ 1.213677e+07# disk./.usageGB‚1173.40970993042# disk.nvme0c0n1.in‚ 21347.6953125) disk.nvme0c0n1.out‚134288.53955078125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1676.3515625 network.recv‚ 1.793415e+06$ disk.dm-0.out‚134288.53955078125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 160.30078125" proc.memory.rssMB‚ 2936.3046875* proc.memory.percent‚17.921781539916992þeG}™‚(:’ ƒüèÔÑÈç® gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚140.918' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.183600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ¾”¢|š‚(:ñ ƒüèÔ–•Õ± network.sent‚ 1.2142093e+07 disk.sda.out‚ 160.6484375! disk.dm-0.out‚134660.83984375 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚89.76016044616699* proc.memory.percent‚17.921781539916992 proc.cpu.threads‚17$ disk.nvme0n1.out‚134660.83984375 network.recv‚ 1.796921e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚134660.83984375 disk.drbd1260.out‚ 0.78515625) disk./.usagePercent‚78.76528815552675% disk./.usageGB‚1173.6597137451172 disk.sda1.in‚0 disk.dm-0.in‚21516.49609375 disk.sda3.in‚0 disk.dm-1.out‚ 160.6484375" proc.memory.rssMB‚ 2936.3046875 cpu‚25.2620343403897$ disk.nvme0c0n1.in‚21516.49609375" disk.nvme0n1.in‚21516.49609375 disk.sda3.out‚ 160.6484375( proc.memory.availableMB‚ 1677.6953125J„ µ2›- _wandbruntime‚5418  _runtime‚5418je˜ÆœÀ $ _timestamp‚1.788493318789658e+09 _runtime‚5418.430027601 _step‚412 & lr-AdamW/pg1‚0.0005727235343595544 & lr-AdamW/pg2‚0.0005727235343595544 trainer/global_step‚5049eæMõËÅ & lr-AdamW/pg1‚0.0005727235343595544 & lr-AdamW/pg2‚0.0005727235343595544 trainer/global_step‚5049 $ _timestamp‚1.788493318789658e+09 _runtime‚5418.430027601 _step‚412œÖ j2ž- _wandbruntime‚5418  _runtime‚5418IØOõ'ŸïO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚353363751071232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392366108385280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5173647342350 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5192340260352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚344312577536624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78913439 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50739777 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28174201 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5415375002624 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5192340260352 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚353363751071232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112092355 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 $ _timestamp‚1.788493318790723e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚358556881979392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚358556091331584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200561053 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112069339 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5193112905216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚397781507328000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5193112905216 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358556091331584 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚353363769074176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88491716 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50739775 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112092355 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358556881979392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392366132325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚344312594575984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200584071 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5415176666112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28173664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 _step‚413>:U ú' ôO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78913439 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50739777 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28174201 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5415375002624 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5192340260352 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚353363751071232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112092355 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 $ _timestamp‚1.788493318790723e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚358556881979392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚358556091331584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200561053 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112069339 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5193112905216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚397781507328000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5193112905216 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358556091331584 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚353363769074176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88491716 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50739775 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112092355 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358556881979392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392366132325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚344312594575984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200584071 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5415176666112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28173664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 _step‚413 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚353363751071232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392366108385280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5173647342350 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5192340260352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚344312577536624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516%Øü±2¡- _wandbruntime‚5419  _runtime‚5419ÁhJ* ¢úM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349553292454612 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200622682 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚344378661011744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _step‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚353431451010560 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚397857666509824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5194117664256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50749425 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5416420687872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5193345006080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5175419130848 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112114092 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78928409 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234846208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚358624796016640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5174648482228 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358625586677760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28179530 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210906112k¯8'£ÿM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78928955 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5174648482228 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358625586677760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28179530 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210906112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349553292454612 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200622682 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚344378661011744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _step‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚353431451010560 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚397857666509824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5194117664256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50749425 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5416420687872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5193345006080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5175419130848 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112114092 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78928409 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234846208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚358624796016640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚361991339998208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89335414 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79660079 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚401600986848256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5467437383680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚401600770863104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113158241 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 P@DeviceStatsMonitor.on_train_batch_start/al™n¡~† located_bytes.all.peak‚ 11892004352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5243364605440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5242591960576 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89335414 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5223717927458 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚356748748037632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚356748766040576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _step‚417 Jû'²õO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5242591960576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚347615916363984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5224488575870 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚361991339998208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89335414 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79660079 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚401600986848256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5467437383680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚401600770863104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113158241 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5243364605440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5242591960576 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89335414 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5223717927458 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚356748748037632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚356748766040576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _step‚417 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79674521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚356816447976960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202532240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5225489715956 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28443225 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚352907472515700 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5244369364480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5468484008960 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚362060835344384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89352290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228554752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _runtime‚5469.023111859 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202555284 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89352288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚362060835344384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5244369364480 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113179952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5243596706304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79675050 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚396208897807360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202532240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚352906684827720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113202994 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202555284 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚347681982799744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5224719067336 % _timestamp‚1.7884933693820133e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89352290 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51231825 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89352288 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 _step‚418 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚401677381816320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396208873867264 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113179952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516áÁ¡ç*#µÿM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚356816447976960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202532240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5225489715956 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28443225 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚352907472515700 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5244369364480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5468484008960 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚362060835344384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89352290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228554752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _runtime‚5469.023111859 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202555284 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89352288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚362060835344384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5244369364480 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113179952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5243596706304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79675050 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚396208897807360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202532240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚352906684827720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113202994 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202555284 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚347681982799744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5224719067336 % _timestamp‚1.7884933693820133e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89352290 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51231825 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89352288 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 _step‚418 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚401677381816320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396208873867264 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113179952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segmeny4z¦Ût.large_pool.current‚516 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113202994 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚356816465979904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51231823 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚401677153261568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 trainer/global_step‚5099 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚356816447976960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79674521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¢<Õ2¶- _wandbruntime‚5469  _runtime‚5469­}ëÍX·Ò 1loss/FBt1/assignment_loss‚0.17229145880420865 0loss/SRqqt3/detection_loss‚1.232168706563803 0loss/FRt3/assignment_loss‚1.9041405069828032 /loss/FRt4/assignment_loss‚1.654464124179468 2loss/SRqqt4/assignment_loss‚1.0619210201929001 /loss/FBt2/assignment_loss‚0.365667188053294 1loss/FBt3/assignment_loss‚0.03640680443495512 1loss/SRqqt2/detection_loss‚2.8334035724401474 /loss/FBt4/detection_loss‚0.5240585122174949 epoch‚4 /loss/FRt3/detection_loss‚1.8765667828917503 1loss/SRqqt4/detection_loss‚1.1330464867445138 %loss/total_loss‚43.17619656425683 _step‚419 .loss/FRt2/detection_loss‚2.704314725749054 0loss/FRt1/assignment_loss‚3.1973586484141974 1loss/SRqqt3/assignment_loss‚1.044065973699057 1loss/FBt4/assignment_loss‚0.13919045066092156 /loss/FRt4/detection_loss‚1.8506330059977567 .loss/FBt1/detection_loss‚2.292107437786303 .loss/FBt2/detection_loss‚2.947979561984539 /loss/FRt2/assignment_loss‚3.806538816860744 .loss/FRt1/detection_loss‚2.227832366105007 /loss/FBt3/detection_loss‚4.2865006685256954 trainer/global_step‚5099 _runtime‚5469.024815618 1loss/SRqqt1/assignment_loss‚1.267015702277422 2loss/SRqqt2/assignment_loss‚1.7289447120080392 % _timestamp‚1.7884933693846064e+09 1loss/SRqqt1/detection_loss‚2.8895793306827544,f]¸× /loss/FBt4/detection_loss‚0.5240585122174949 epoch‚4 /loss/FRt3/detection_loss‚1.8765667828917503 1loss/SRqqt4/detection_loss‚1.1330464867445138 %loss/total_loss‚43.17619656425683 _step‚419 .loss/FRt2/detection_loss‚2.704314725749054 0loss/FRt1/assignment_loss‚3.1973586484141974 1loss/SRqqt3/assignment_loss‚1.044065973699057 1loss/FBt4/assignment_loss‚0.13919045066092156 /loss/FRt4/detection_loss‚1.8506330059977567 .loss/FBt1/detection_loss‚2.292107437786303 .loss/FBt2/detection_loss‚2.947979561984539 /loss/FRt2/assignment_loss‚3.806538816860744 .loss/FRt1/detection_loss‚2.227832366105007 /loss/FBt3/detection_loss‚4.2865006685256954 trainer/global_step‚5099 _runtime‚5469.024815618 1loss/SRqqt1/assignment_loss‚1.267015702277422 2loss/SRqqt2/assignment_loss‚1.7289447120080392 % _timestamp‚1.7884933693846064e+09 1loss/SRqqt1/detection_loss‚2.8895793306827544 1loss/FBt1/assignment_loss‚0.17229145880420865 0loss/SRqqt3/detection_loss‚1.232168706563803 0loss/FRt3/assignment_loss‚1.9041405069828032 /loss/FRt4/assignment_loss‚1.654464124179468 2loss/SRqqt4/assignment_loss‚1.0619210201929001 /loss/FBt2/assignment_loss‚0.365667188053294 1loss/FBt3/assignment_loss‚0.03640680443495512 1loss/SRqqt2/detection_loss‚2.8334035724401474£AŸ8¦¹‚(:… ¿üèÔº·®® gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.92' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³i'®ƒº‚(:ø ¿üèԬί disk.drbd1260.in‚0( proc.memory.availableMB‚ 1677.8359375 network.recv‚ 1.815335e+06) disk./.usagePercent‚78.78294934729139 disk.sda2.out‚0) disk.nvme0c0n1.out‚135728.42138671875 disk.dm-1.out‚ 162.296875" proc.memory.rssMB‚ 2936.3046875 proc.cpu.threads‚17 disk.sda1.in‚0" disk.nvme0n1.in‚21653.98046875' disk.nvme0n1.out‚135728.42138671875$ disk.dm-0.out‚135728.42138671875 disk.sda3.out‚ 162.296875 disk.drbd1260.out‚ 0.79296875 disk.sda.in‚0$ memory_percent‚89.75930213928223 disk.sda2.in‚0 disk.dm-0.in‚21653.98046875 disk.sda.out‚ 162.296875* proc.memory.percent‚17.921781539916992 cpu‚25.26021851243531 network.sent‚ 1.2267308e+07% disk./.usageGB‚1173.9228782653809 disk.sda1.out‚0$ disk.nvme0c0n1.in‚21653.98046875 disk.sda3.in‚0 disk.dm-1.in‚0ŠWS»‚(:… ÎüèÔô†Ò® gpu.0.gpu‚48.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.41' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.282 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)_CUu¼‚(:ê ÎüèÔÝÌ×° disk.sda.in‚0 disk.sda.out‚162.75" disk.nvme0n1.in‚21653.98046875 disk.sda3.out‚162.75 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0n1.out‚136214.85498046875' proc.memory.availableMB‚ 1569.296875" proc.memory.rssMB‚ 2936.3046875 proc.cpu.threads‚17 disk.dm-1.out‚162.75 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚136214.85498046875 disk.dm-0.in‚21653.98046875$ disk.dm-0.out‚136214.85498046875* proc.memory.percent‚17.921781539916992 network.recv‚ 1.821289e+06 disk.drbd1260.out‚ 0.79296875 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚21653.98046875$ memory_percent‚90.42177200317383 cpu‚25.259727072255828 network.sent‚ 1.2377276e+07( disk./.usagePercent‚78.7912555117398$ disk./.usageGB‚1174.046646118164 disk.sda3.in‚0 disk.dm-1.in‚0‚ä{?’½‚(:‡ ÝüèÔùŽË­ gpu.0.gpu‚37.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.563' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9126 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U¨b|¾‚(:ñ ÝüèÔ‡ö’° cpu‚25.259303089626343 proc.cpu.threads‚17% disk./.usageGB‚1174.1938972473145$ disk.nvme0c0n1.in‚21653.98046875" disk.nvme0n1.in‚21653.98046875 disk.dm-1.out‚163.3125 disk.drbd1260.in‚0) proc.memory.availableMB‚ 1481.07421875" proc.memory.rssMB‚ 2936.3046875 network.recv‚ 1.825774e+06 disk.sda.in‚0 disk.sda.out‚163.3125 disk.sda1.out‚0' disk.nvme0n1.out‚136659.63330078125$ disk.dm-0.out‚136659.63330078125 disk.sda3.out‚163.3125 disk.drbd1260.out‚0.796875 network.sent‚ 1.2383463e+07 disk.sda1.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚136659.63330078125 disk.dm-0.in‚21653.98046875 disk.sda3.in‚0# memory_percent‚90.9602403640747* proc.memory.percent‚17.921781539916992) disk./.usagePercent‚78.80113765856902 disk.sda2.in‚0 disk.dm-1.in‚0a’2z’¿‚(:‡ ìüèÔ—Ÿ”± gpu.0.gpu‚38.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.558' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.1116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v}t…À‚(:ú ìüèÔ´°ã²$ disk.nvme0c0n1.in‚21653.98046875) disk.nvme0c0n1.out‚137001.34814453125) disk./.usagePercent‚78.80114457077244 disk.sda1.in‚0' disk.nvme0n1.out‚137001.34814453125$ disk.dm-0.out‚137001.34814453125 disk.sda3.out‚ 163.7734375$ memory_percent‚91.59791469573975) proc.memory.availableMB‚ 1376.59765625* proc.memory.percent‚17.921781539916992 disk.sda2.in‚0 disk.dm-1.out‚ 163.7734375 disk.sda.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1174.1940002441406 disk.dm-1.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚21653.98046875 disk.dm-0.in‚21653.98046875 disk.sda3.in‚0" proc.memory.rssMB‚ 2936.3046875 cpu‚25.25894158017755 network.sent‚ 1.2387616e+07 network.recv‚ 1.830299e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚0.796875 disk.sda.out‚ 163.7734375 disk.sda1.out‚0.—'“2Á- _wandbruntime‚5520  _runtime‚5520M¨,=ÇÂÁ % _timestamp‚1.7884934206546288e+09 _runtime‚5520.295254245 _step‚420 & lr-AdamW/pg1‚0.0005624139639017577 & lr-AdamW/pg2‚0.0005624139639017577 trainer/global_step‚5149o[ÏwÌÃÆ & lr-AdamW/pg1‚0.0005624139639017577 & lr-AdamW/pg2‚0.0005624139639017577 trainer/global_step‚5149 % _timestamp‚1.7884934206546288e+09 _runtime‚5520.295254245 _step‚420¤²ŒL2Ä- _wandbruntime‚5520  _runtime‚5520œ¢Ë˜õ'ÅïO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365426588664832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚360133763006976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360133745004032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204426257 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28700638 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356193797312962 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚421 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51704575 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399900942663680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90179116 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚365427379312640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚405420263105536 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚5149 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5273788512566 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5519320441856 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚405420491673600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 $ _timestamp‚1.788493420656402e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5292843660800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360133745004032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80405215 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28700095 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5274559160978 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5292843660800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204449275 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90179116 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80404670 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚365427379312640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5519320441856 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚405420491673600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 $ _timestamp‚1.788493420656402e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5292843660800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360133745004032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80405215 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28700095 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5274559160978 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5292843660800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204449275 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90179116 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80404670 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚365427379312640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365426588664832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚360133763006976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360133745004032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204426257 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28700638 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356193797312962 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚421 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0¥ru2Ç- _wandbruntime‚5521  _runtime‚5521ÙàFF] ÈúM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51714225 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405496663222272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204464842 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204464842 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28705324 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399976291006464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204487886 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114291896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80419547 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51714223 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28705848 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80420073 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 JBDeviceStatsMonitor.on_train_batch_end/segment.smalÙž¥£l_pool.allocated‚489 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884934218601828e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114268854 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405496881291264 trainer/global_step‚5149 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194128896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5275560301064 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚365496084011008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90195990 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5293848406528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚350985304587744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5520372215808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚360201462946304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114268854 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399976314946560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚350985287548384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5293848406528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5294621064704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204487886 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356260864888808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5520566344704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚422 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360201444943360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 _runtime‚5521.501415652 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114291896þu'ÉÿM BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚350985304587744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5520372215808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚360201462946304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114268854 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399976314946560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚350985287548384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5293848406528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5294621064704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204487886 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356260864888808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5520566344704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚422 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360201444943360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 _runtime‚5521.501415652 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114291896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51714225 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405496663222272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204464842 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204464842 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28705324 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399976291006464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204487886 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114291896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80419547 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51714223 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28705848 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80420073 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884934218601828e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114268854 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405496881291264 trainer/global_step‚5149 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194128896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5275560301064 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚365496084011008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90195990 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5293848406528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005¦”†9j2Ê- _wandbruntime‚5521  _runtime‚5521ð÷H1WËÑ 0loss/FBt1/assignment_loss‚0.3090500370062028 /loss/FRt4/detection_loss‚1.4604407763987217 1loss/SRqqt3/detection_loss‚1.4895101248378484 1loss/SRqqt1/detection_loss‚2.0687789345109784 trainer/global_step‚5149 % _timestamp‚1.7884934218633394e+09 /loss/FRt2/assignment_loss‚3.854009084403515 .loss/FRt4/assignment_loss‚1.41252534681896 2loss/SRqqt1/assignment_loss‚1.5718593756492074 epoch‚4 %loss/total_loss‚40.20851587775047 0loss/FBt3/assignment_loss‚0.3487823342147749 /loss/FRt1/detection_loss‚2.5262531876564025 0loss/SRqqt2/detection_loss‚2.649673322072396 _step‚423 0loss/FRt1/assignment_loss‚2.9769808689753217 1loss/FBt4/assignment_loss‚0.18091194548482092 /loss/FRt2/detection_loss‚2.6869083195924977 /loss/FBt4/detection_loss‚0.4896354887908434 1loss/SRqqt4/assignment_loss‚1.111854148556439 0loss/FBt2/assignment_loss‚0.5223805870433958 .loss/FRt3/detection_loss‚2.201128742269551 0loss/SRqqt4/detection_loss‚1.068217962665457 2loss/SRqqt2/assignment_loss‚1.4399912082231963 2loss/SRqqt3/assignment_loss‚1.1239026599550541 /loss/FBt1/detection_loss‚1.7846799767934358 .loss/FBt2/detection_loss‚2.675971736510595 /loss/FBt3/detection_loss‚2.1999494671821593 _runtime‚5521.503597011 /loss/FRt3/assignment_loss‚2.055120242138704dRÖð\ÌÖ 0loss/SRqqt4/detection_loss‚1.068217962665457 2loss/SRqqt2/assignment_loss‚1.4399912082231963 2loss/SRqqt3/assignment_loss‚1.1239026599550541 /loss/FBt1/detection_loss‚1.7846799767934358 .loss/FBt2/detection_loss‚2.675971736510595 /loss/FBt3/detection_loss‚2.1999494671821593 _runtime‚5521.503597011 /loss/FRt3/assignment_loss‚2.055120242138704 0loss/FBt1/assignment_loss‚0.3090500370062028 /loss/FRt4/detection_loss‚1.4604407763987217 1loss/SRqqt3/detection_loss‚1.4895101248378484 1loss/SRqqt1/detection_loss‚2.0687789345109784 trainer/global_step‚5149 % _timestamp‚1.7884934218633394e+09 /loss/FRt2/assignment_loss‚3.854009084403515 .loss/FRt4/assignment_loss‚1.41252534681896 2loss/SRqqt1/assignment_loss‚1.5718593756492074 epoch‚4 %loss/total_loss‚40.20851587775047 0loss/FBt3/assignment_loss‚0.3487823342147749 /loss/FRt1/detection_loss‚2.5262531876564025 0loss/SRqqt2/detection_loss‚2.649673322072396 _step‚423 0loss/FRt1/assignment_loss‚2.9769808689753217 1loss/FBt4/assignment_loss‚0.18091194548482092 /loss/FRt2/detection_loss‚2.6869083195924977 /loss/FBt4/detection_loss‚0.4896354887908434 1loss/SRqqt4/assignment_loss‚1.111854148556439 0loss/FBt2/assignment_loss‚0.5223805870433958 .loss/FRt3/detection_loss‚2.201128742269551§ÕâêmÍ‚(:’ ûüèÔÔÒõ­ gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.069' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.213800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÓšS‰Î‚(:þ ûüèÔÄÙâ® disk.sda2.out‚0% disk./.usageGB‚1173.7222709655762 disk.sda3.out‚ 164.26171875 disk.sda.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚21778.60546875) disk.nvme0c0n1.out‚137205.21142578125* proc.memory.percent‚17.921781539916992 cpu‚25.258464872704334 network.recv‚ 1.833196e+06$ disk.dm-0.out‚137205.21142578125 disk.sda3.in‚0 disk.sda.out‚ 164.26171875 disk.sda1.in‚0" proc.memory.rssMB‚ 2936.3046875 proc.cpu.threads‚17" disk.nvme0n1.in‚21778.60546875 disk.dm-1.out‚ 164.26171875 disk.drbd1260.out‚ 0.80078125$ memory_percent‚92.18287467956543( proc.memory.availableMB‚ 1280.7578125 disk.drbd1260.in‚0 network.sent‚ 1.239169e+07) disk./.usagePercent‚78.76948642308076' disk.nvme0n1.out‚137205.21142578125 disk.dm-0.in‚21778.60546875 disk.dm-1.in‚0 disk.sda1.out‚0l ’Ï‚(:‡ ŠýèÔ— ¤¯ gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.827' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N!:ЇÐ‚(:ü ŠýèÔʹԳ% disk./.usageGB‚1173.7085189819336 disk.sda3.in‚0 disk.sda3.out‚ 164.6484375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚21778.60546875 network.sent‚ 1.2499601e+07 network.recv‚ 1.839012e+06) disk./.usagePercent‚78.76856351591982 disk.dm-1.out‚ 164.6484375 disk.drbd1260.out‚ 0.80078125 disk.sda.out‚ 164.6484375" proc.memory.rssMB‚ 2936.3046875* proc.memory.percent‚17.921781539916992 disk.sda2.out‚0) disk.nvme0c0n1.out‚137459.09814453125 disk.dm-0.in‚21778.60546875$ disk.dm-0.out‚137459.09814453125 disk.drbd1260.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚21778.60546875' disk.nvme0n1.out‚137459.09814453125$ memory_percent‚92.87045001983643) proc.memory.availableMB‚ 1168.10546875 cpu‚25.25808627244154 proc.cpu.threads‚17œ'ü+’Ñ‚(:‡ ™ýèÔöëÕ® gpu.0.gpu‚15.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.256' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃMD½‡Ò‚(:ü ™ýèÔ€«ü° proc.cpu.threads‚17% disk./.usageGB‚1173.7093544006348 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8046875 disk.sda.out‚ 165.0703125 disk.sda2.out‚0$ disk.nvme0c0n1.in‚21778.60546875) disk.nvme0c0n1.out‚137832.93603515625' disk.nvme0n1.out‚137832.93603515625 network.sent‚ 1.2505885e+07 disk.dm-1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚21778.60546875 disk.sda3.in‚0" proc.memory.rssMB‚ 2936.3046875* proc.memory.percent‚17.921781539916992 cpu‚25.257646707640763 network.recv‚ 1.843762e+06 disk.sda.in‚0 disk.sda2.in‚0$ disk.dm-0.out‚137832.93603515625$ memory_percent‚90.05506038665771) proc.memory.availableMB‚ 1629.37890625) disk./.usagePercent‚78.76861958156982 disk.dm-1.out‚ 165.0703125 disk.sda1.in‚0" disk.nvme0n1.in‚21778.60546875 disk.sda3.out‚ 165.0703125T` >2Ó- _wandbruntime‚5570  _runtime‚5570´(2ÇÇÔÁ & lr-AdamW/pg2‚0.0005572066568966899 trainer/global_step‚5199 % _timestamp‚1.7884934710148215e+09 _runtime‚5570.655396641 _step‚424 & lr-AdamW/pg1‚0.0005572066568966899ˆ %ÌÕÆ & lr-AdamW/pg1‚0.0005572066568966899 & lr-AdamW/pg2‚0.0005572066568966899 trainer/global_step‚5199 % _timestamp‚1.7884934710148215e+09 _runtime‚5570.655396641 _step‚424¨Èù¦á2Ö- _wandbruntime‚5570  _runtime‚5570Øü,J´×ïO trainer/global_step‚5199 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28963826 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚517 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409240038529536 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368862627979264 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368862627979264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52186977 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115336045 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91022816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52186975 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5571864480256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚359547189686070 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115336045 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363518741970432 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363518741970432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81150284 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91022816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚354222542900624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5343868005888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185753600 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91022814 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403668359802880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5571678726656 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28963309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91022814 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5343868005888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81150803 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115359061 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209693696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 _step‚425 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368861837331456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5343095361024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206381877 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5323859097674 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403668383742976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚5570.656709835 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206358859 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 $ _timestamp‚1.788493471016074e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5343095361024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363518759973376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363518741970432 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363518741970432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81150284 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91022816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚354222542900624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5343868005888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185753600 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91022814 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403668359802880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5571678726656 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28963309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91022814 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5343868005888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81150803 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115359061 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209693696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 _step‚425 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368861837331456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5343095361024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206381877 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5323859097674 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403668383742976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚5570.656709835 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206358859 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 $ _timestamp‚1.788493471016074e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5343095361024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363518759973376 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207583232 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 206397444 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115357756 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363586459912704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403743708145664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5572744760832 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91039688 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5344872764928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81165187 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354288626375744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115357756 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91039688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354288609336384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5344100106752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚5199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5325630886172 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5572928403968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359613469573936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5344872764928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81165715 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28969090 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409316452906496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183643136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _step‚426 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403743732085760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91039690 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206420488 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359614257261916 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚5571.562071199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚368931332677632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206397444 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115380798 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363586441909760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368931332677632 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28968564 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 206420488 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368930542016512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207583232 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 206397444 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115357756 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363586459912704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403743708145664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5572744760832 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91039688 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5344872764928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81165187 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354288626375744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115357756 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91039688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354288609336384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5344100106752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚5199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5325630886172 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5572928403968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359613469573936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5344872764928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81165715 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28969090 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409316452906496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183643136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _step‚426 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403743732085760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91039690 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206420488 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359614257261916 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚5571.562071199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚368931332677632ªaÇà—2Ü- _wandbruntime‚5571  _runtime‚5571Þ(ž+Ý× 0loss/FRt1/assignment_loss‚2.9852767226307892 1loss/SRqqt4/assignment_loss‚1.187459102237894 0loss/FBt1/assignment_loss‚0.1396688832235538 1loss/SRqqt1/detection_loss‚2.2636495920327993 1loss/SRqqt3/detection_loss‚1.2237303441183434 -loss/FBt1/detection_loss‚3.1145867í€y#21944482 /loss/FBt3/detection_loss‚3.7751159071922302 /loss/FBt4/detection_loss‚0.5503208925688846 2loss/SRqqt2/assignment_loss‚2.0237102723783917 /loss/FRt4/detection_loss‚1.5446345879042402 1loss/SRqqt2/detection_loss‚2.9416584405634136 trainer/global_step‚5199 % _timestamp‚1.7884934719233623e+09 _step‚427 1loss/FBt2/assignment_loss‚0.09409652947479116 0loss/FBt4/assignment_loss‚0.1351210685709451 -loss/FRt1/detection_loss‚2.26652204600531 /loss/FRt3/detection_loss‚2.2646824369827905 1loss/SRqqt4/detection_loss‚1.1062088025652843 _runtime‚5571.563517303 0loss/FRt3/assignment_loss‚1.6983703209294214 0loss/FRt4/assignment_loss‚1.2632961311238877 1loss/SRqqt1/assignment_loss‚1.680570909466881 /loss/FRt2/detection_loss‚2.2564637065847135 2loss/SRqqt3/assignment_loss‚1.1761484228902392 /loss/FBt2/detection_loss‚2.8996240139007567 epoch‚4 0loss/FRt2/assignment_loss‚3.6625806032082973 3loss/FBt3/assignment_loss‚0.0004840908950427547 %loss/total_loss‚42.25398054689372™ËTbÞÜ 0loss/FRt3/assignment_loss‚1.6983703209294214 0loss/FRt4/assignment_loss‚1.2632961311238877 1loss/SRqqt1/assignment_loss‚1.680570909466881 /loss/FRt2/detection_loss‚2.2564637065847135 2loss/SRqqt3/assignment_loss‚1.1761484228902392 /loss/FBt2/detection_loss‚2.8996240139007567 epoch‚4 0loss/FRt2/assignment_loss‚3.6625806032082973 3loss/FBt3/assignment_loss‚0.0004840908950427547 %loss/total_loss‚42.25398054689372 0loss/FRt1/assignment_loss‚2.9852767226307892 1loss/SRqqt4/assignment_loss‚1.187459102237894 0loss/FBt1/assignment_loss‚0.1396688832235538 1loss/SRqqt1/detection_loss‚2.2636495920327993 1loss/SRqqt3/detection_loss‚1.2237303441183434 -loss/FBt1/detection_loss‚3.11458671944482 /loss/FBt3/detection_loss‚3.7751159071922302 /loss/FBt4/detection_loss‚0.5503208925688846 2loss/SRqqt2/assignment_loss‚2.0237102723783917 /loss/FRt4/detection_loss‚1.5446345879042402 1loss/SRqqt2/detection_loss‚2.9416584405634136 trainer/global_step‚5199 % _timestamp‚1.7884934719233623e+09 _step‚427 1loss/FBt2/assignment_loss‚0.09409652947479116 0loss/FBt4/assignment_loss‚0.1351210685709451 -loss/FRt1/detection_loss‚2.26652204600531 /loss/FRt3/detection_loss‚2.2646824369827905 1loss/SRqqt4/detection_loss‚1.1062088025652843 _runtime‚5571.563517303«9”ß’ß‚(:‡ ¨ýèÔý§ì® gpu.0.gpu‚50.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.041' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+-ë<{à‚(:ð ¨ýèÔ´Ÿê¯ network.sent‚ 1.251165e+07 disk.drbd1260.out‚ 0.8046875 disk.sda.out‚165.5625 disk.sda2.in‚0$ disk.nvme0c0n1.in‚21778.60546875" disk.nvme0n1.in‚21778.60546875' disk.nvme0n1.out‚137982.55322265625 disk.dm-0.in‚21778.60546875 network.recv‚ 1.847797e+06 disk.dm-1.in‚0 disk.dm-1.out‚165.5625 disk.sda1.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚137982.55322265625" proc.memory.rssMB‚ 2936.3046875( disk./.usagePercent‚78.7686823034157% disk./.usageGB‚1173.7102890014648 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚165.5625$ memory_percent‚90.71753025054932) proc.memory.availableMB‚ 1520.83984375* proc.memory.percent‚17.921781539916992 disk.drbd1260.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚137982.55322265625 cpu‚25.25723936241327 proc.cpu.threads‚17 o–O’á‚(:‡ ·ýèÔ‚ê“­ gpu.0.gpu‚39.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.748' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ș ìvâ‚(:ë ·ýèÔƒéþ¯ disk.dm-1.in‚0 disk.dm-1.out‚ 166.0546875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.80859375 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 21788.5625 network.sent‚ 1.2619989e+07 network.recv‚ 1.853883e+06& disk.nvme0n1.out‚138422.2255859375# disk.dm-0.out‚138422.2255859375" proc.memory.rssMB‚ 2936.3046875* proc.memory.percent‚17.921781539916992 proc.cpu.threads‚17( disk./.usagePercent‚78.7680146357664 disk.sda.in‚0 disk.sda.out‚ 166.0546875 disk.sda1.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 21788.5625) proc.memory.availableMB‚ 1416.61328125 cpu‚25.256822365523185$ disk./.usageGB‚1173.700340270996 disk.sda3.out‚ 166.0546875( disk.nvme0c0n1.out‚138422.2255859375 disk.dm-0.in‚ 21788.5625# memory_percent‚91.3536787033081 disk.sda3.in‚03†Ðe’ã‚(:‡ ÆýèÔÊΈ­ gpu.0.gpu‚43.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚122.504' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö´­Ÿ|ä‚(:ñ ÆýèÔÎŽ° network.sent‚ 1.2624847e+07 disk.sda3.out‚ 166.765625 disk.drbd1260.out‚ 0.80859375 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 21905.2421875# disk.dm-0.out‚138854.9716796875 cpu‚25.25640786763645 disk.sda3.in‚0 disk.sda.out‚ 166.765625 disk.sda1.out‚0& disk.nvme0n1.out‚138854.9716796875$ memory_percent‚92.03896522521973* proc.memory.percent‚17.921781539916992 network.recv‚ 1.85714e+06) disk./.usagePercent‚78.74696160018063 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚138854.9716796875 disk.dm-0.in‚ 21905.2421875% disk./.usageGB‚1173.3866348266602 disk.dm-1.out‚ 166.765625 disk.sda.in‚0! disk.nvme0n1.in‚ 21905.2421875( proc.memory.availableMB‚ 1304.3359375" proc.memory.rssMB‚ 2936.3046875 proc.cpu.threads‚17s·Ã2å- _wandbruntime‚5619  _runtime‚5619L öæÆæÀ & lr-AdamW/pg2‚0.0005519657798934455 trainer/global_step‚5249 $ _timestamp‚1.788493520204219e+09 _runtime‚5619.844527367 _step‚428 & lr-AdamW/pg1‚0.0005519657798934455Ä•5ŠËçÅ _runtime‚5619.844527367 _step‚428 & lr-AdamW/pg1‚0.0005519657798934455 & lr-AdamW/pg2‚0.0005519657798934455 trainer/global_step‚5249 $ _timestamp‚1.788493520204219e+09¬ñüû½2è- _wandbruntime‚5619  _runtime‚5619ENãö'éðO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362900582059178 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208314479 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚357525881727984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208291461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29228308 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81897685 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚413059904107008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5624103224832 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81897131 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52669375 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362899794371406 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚366903756939776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407435800882176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5374700331194 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚413059683927552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5623906985472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚372297876645888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5393347061248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357525864688624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372297085998080 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚372297085998080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 trainer/global_step‚5249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116447963 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚413059904107008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5624103224832 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81897131 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52669375 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362899794371406 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚366903756939776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407435800882176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5374700331194 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚413059683927552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5623906985472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚372297876645888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5393347061248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357525864688624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372297085998080 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚372297085998080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 trainer/global_step‚5249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116447963 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362966861947044 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚372366581344256 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208353090 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5375701471280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116469700 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407511149224960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5625153340416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5624955017216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116446658 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚357591948163744 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116446658 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208353090 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5394351806976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81912606 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366971456879104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚366971438876160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5394351806976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91883388 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52679023 trainer/global_step‚5249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91883390 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91883390 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.frˆ-Â>eed‚0 $ _timestamp‚1.788493521042422e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚430 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29233581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5395124465152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372365790683136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208330046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚372366581344256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357591931124384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5395124465152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81912071 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91883388 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362967649635024 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚413136080302080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980/;Ô'íþM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚430 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29233581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5395124465152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372365790683136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208330046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚372366581344256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357591931124384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5395124465152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81912071 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91883388 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362967649635024 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚413136080302080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29233048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407511125284864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362966861947044 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚372366581344256 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208353090 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5375701471280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116469700 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407511149224960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5625153340416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5624955017216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116446658 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚357591948163744 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116446658 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208353090 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5394351806976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81912606 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366971456879104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚366971438876160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5394351806976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91883388 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52679023 trainer/global_step‚5249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91883390 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91883390 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788493521042422e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533®i–šù2î- _wandbruntime‚5620  _runtime‚5620˜J[ïÕ 0loss/FRt2/assignment_loss‚3.3042240651214825 /loss/FRt1/detection_loss‚2.4833638262199726 .loss/FBt3/detection_loss‚2.116205174814571 epoch‚4 trainer/global_step‚5249 _runtime‚5620.685124013 0loss/FRt4/assignment_loss‚1.2744084173982793 .loss/FRt2/detection_loss‚2.383929282472038 1loss/SRqqt4/detection_loss‚0.8992721290445663 %loss/total_loss‚38.82530078925756 0loss/FRt3/assignment_loss‚1.7917974775745755 1loss/FBt3/assignment_loss‚0.23070293833271452 /loss/FRt3/detection_loss‚2.3494869028883323 0loss/SRqqt1/detection_loss‚2.392565542459488 _step‚431 1loss/SRqqt1/assignment_loss‚1.452615150809288 1loss/SRqqt3/assignment_loss‚1.148154236599803 .loss/FBt2/detection_loss‚2.506662896701268 1loss/FBt4/assignment_loss‚0.14242489130652664 0loss/FRt1/assignment_loss‚2.9882450919402275 2loss/FBt1/assignment_loss‚0.015955454427360866 /loss/FRt4/detection_loss‚1.5707178620452231 1loss/SRqqt3/detection_loss‚1.1874780216316383 .loss/FBt4/detection_loss‚0.513266783580184 1loss/SRqqt2/assignment_loss‚1.694901832845062 2loss/SRqqt4/assignment_loss‚0.9550076719950622 1loss/FBt2/assignment_loss‚0.21030630415738674 0loss/SRqqt2/detection_loss‚2.222054783254862 /loss/FBt1/detection_loss‚2.9915540516376495 % _timestamp‚1.7884935210449538e+09n#®ý`ðÚ 0loss/FRt1/assignment_loss‚2.9882450919402275 2loss/FBt1/assignment_loss‚0.015955454427360866 /loss/FRt4/detection_loss‚1.5707178620452231 1loss/SRqqt3/detection_loss‚1.1874780216316383 .loss/FBt4/detection_loss‚0.513266783580184 1loss/SRqqt2/assignment_loss‚1.694901832845062 2loss/SRqqt4/assignment_loss‚0.9550076719950622 1loss/FBt2/assignment_loss‚0.21030630415738674 0loss/SRqqt2/detection_loss‚2.222054783254862 /loss/FBt1/detection_loss‚2.9915540516376495 % _timestamp‚1.7884935210449538e+09 0loss/FRt2/assignment_loss‚3.3042240651214825 /loss/FRt1/detection_loss‚2.4833638262199726 .loss/FBt3/detection_loss‚2.116205174814571 epoch‚4 trainer/global_step‚5249 _runtime‚5620.685124013 0loss/FRt4/assignment_loss‚1.2744084173982793 .loss/FRt2/detection_loss‚2.383929282472038 1loss/SRqqt4/detection_loss‚0.8992721290445663 %loss/total_loss‚38.82530078925756 0loss/FRt3/assignment_loss‚1.7917974775745755 1loss/FBt3/assignment_loss‚0.23070293833271452 /loss/FRt3/detection_loss‚2.3494869028883323 0loss/SRqqt1/detection_loss‚2.392565542459488 _step‚431 1loss/SRqqt1/assignment_loss‚1.452615150809288 1loss/SRqqt3/assignment_loss‚1.148154236599803 .loss/FBt2/detection_loss‚2.506662896701268 1loss/FBt4/assignment_loss‚0.14242489130652664¯o¼É’ñ‚(:‡ ÕýèÔø‡‰± gpu.0.gpu‚34.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËL—ƒò‚(:ø ÕýèÔÙùÛ² disk.dm-1.out‚ 167.23828125$ memory_percent‚92.64938831329346* proc.memory.percent‚17.921781539916992) disk.nvme0c0n1.out‚139047.88720703125% disk./.usageGB‚1173.6371383666992 disk.sda2.in‚0) proc.memory.availableMB‚ 1204.32421875" proc.memory.rssMB‚ 2936.3046875 cpu‚25.25592870530605 proc.cpu.threads‚17 network.sent‚ 1.2631205e+07) disk./.usagePercent‚78.76377310293911 disk.sda.out‚ 167.23828125 disk.sda1.in‚0! disk.nvme0n1.in‚ 21905.2421875 disk.sda3.in‚0 network.recv‚ 1.861969e+06 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 21905.2421875' disk.nvme0n1.out‚139047.88720703125 disk.sda3.out‚ 167.23828125 disk.drbd1260.out‚0.8125 disk.dm-0.in‚ 21905.2421875$ disk.dm-0.out‚139047.88720703125 disk.dm-1.in‚0üIn®’ó‚(:‡ äýèÔ¢ñ‹­ gpu.0.gpu‚37.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.573' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5146 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{;ó}ô‚(:ò äýèÔëㆯ disk.sda.in‚0 disk.sda2.in‚0$ memory_percent‚93.19932460784912 disk.sda3.out‚ 167.6796875 disk.sda2.out‚0* proc.memory.percent‚17.921781539916992 cpu‚25.25546613723578 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 21905.2421875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 167.6796875 disk.sda.out‚ 167.6796875) proc.memory.availableMB‚ 1114.22265625" proc.memory.rssMB‚ 2936.3046875 network.sent‚ 1.274007e+07 network.recv‚ 1.868769e+06( disk./.usagePercent‚78.7621326066597$ disk./.usageGB‚1173.612693786621 disk.dm-0.in‚ 21905.2421875 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚139384.67626953125! disk.nvme0n1.in‚ 21905.2421875' disk.nvme0n1.out‚139384.67626953125$ disk.dm-0.out‚139384.67626953125 disk.drbd1260.out‚0.8125nXTîõ‚(:’ óýèÔ¨õƯ gpu.0.gpu‚28.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.398' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.879599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zàXq|ö‚(:ñ óýèÔÚ›ü± proc.cpu.threads‚17% disk./.usageGB‚1173.6117820739746 disk.drbd1260.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚139769.869140625! disk.nvme0n1.in‚ 21905.2421875$ proc.memory.availableMB‚1118.875 network.recv‚ 1.872316e+06 disk.sda3.in‚0 disk.sda3.out‚ 168.10546875 disk.dm-1.in‚0 disk.dm-1.out‚ 168.10546875 disk.sda.in‚0# disk.nvme0c0n1.in‚ 21905.2421875) disk./.usagePercent‚78.76207142085902 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0% disk.nvme0n1.out‚139769.869140625 disk.dm-0.in‚ 21905.2421875" disk.dm-0.out‚139769.869140625" proc.memory.rssMB‚ 2936.3046875 disk.sda.out‚ 168.10546875 network.sent‚ 1.2745392e+07 disk.drbd1260.out‚ 0.81640625$ memory_percent‚93.17092895507812* proc.memory.percent‚17.921781539916992 cpu‚25.25507669105797¢AZÁ’÷‚(:‡ ‚þèÔÛªú¬ gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.176' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8352 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ü¸'zø‚(:ï ‚þèÔùÕ® disk.sda2.out‚0" proc.memory.rssMB‚ 2936.3046875 disk.sda3.out‚ 168.546875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 168.546875 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 22052.9921875! disk.nvme0n1.in‚ 22052.9921875) disk./.usagePercent‚78.75579770022493 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚140155.642578125% disk.nvme0n1.out‚140155.642578125 disk.dm-0.in‚ 22052.9921875" disk.dm-0.out‚140155.642578125$ memory_percent‚93.17243099212646) proc.memory.availableMB‚ 1118.62890625% disk./.usageGB‚1173.5182991027832 disk.sda3.in‚0 disk.dm-1.out‚ 168.546875 disk.sda2.in‚0* proc.memory.percent‚17.921781539916992 cpu‚25.2547454734527 proc.cpu.threads‚17 network.sent‚ 1.2750318e+07 network.recv‚ 1.875761e+06 disk.drbd1260.out‚ 0.81640625ø‹2ù- _wandbruntime‚5670  _runtime‚5670¨çyWÇúÁ & lr-AdamW/pg1‚0.0005466924250829368 & lr-AdamW/pg2‚0.0005466924250829368 trainer/global_step‚5299 % _timestamp‚1.7884935703842273e+09 _runtime‚5670.024549935 _step‚432 1WºÌûÆ & lr-AdamW/pg1‚0.0005466924250829368 & lr-AdamW/pg2‚0.0005466924250829368 trainer/global_step‚5299 % _timestamp‚1.7884935703842273e+09 _runtime‚5670.024549935 _step‚432°a°T2ü- _wandbruntime‚5670  _runtime‚5670åV0Ä%ýïO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92710214 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚416879339856896 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5676335723520 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117513849 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82643555 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411203218021376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚370288735903232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29492322 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5443598761472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117536865 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210224063 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92710216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117536865 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚360829186476624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚375733125312512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5444371406336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53151775 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5443598761472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚375732334664704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29491780 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5676145775616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 10†‹gâ25507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 trainer/global_step‚5299 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5444371406336 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366253186744514 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5424000267890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576¨üáú'þôO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 trainer/global_step‚5299 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5444371406336 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366253186744514 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5424000267890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92710214 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚416879339856896 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5676335723520 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117513849 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82643555 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411203218021376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚370288735903232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29492322 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5443598761472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117536865 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210224063 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92710216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117536865 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚360829186476624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚375733125312512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5444371406336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53151775 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5443598761472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚375732334664704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29491780 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5676145775616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0±g)2ÿ- _wandbruntime‚5670  _runtime‚5670Q>ÚŠÿ&€ùM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210285692 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5425772056388 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92727090 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92727088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚375801830010880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5677191109632 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117558602 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚555 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82658436 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117535560 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 trainer/global_step‚5299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210262648 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53161425 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5425001407768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5444603507200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5444603507200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210262648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117558602 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5677387335680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53161423 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29497566 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚416955953699840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92727088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚360895252912384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚370356435842560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82658991 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375801830010880 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210285692 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚370356453845504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5445376165376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117535560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚370356453845504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366320254320152#¾)Ž'þM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82658991 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375801830010880 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210285692 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚370356453845504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5445376165376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117535560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚370356453845504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366320254320152 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210285692 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5425772056388 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92727090 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92727088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚375801830010880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5677191109632 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117558602 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚555 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82658436 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117535560 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 trainer/global_step‚5299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210262648 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53161425 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5425001407768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5444603507200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5444603507200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210262648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117558602 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5677387335680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53161423 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29497566 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚416955953699840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92727088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚360895252912384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚370356435842560²pÎj[2‚- _wandbruntime‚5670  _runtime‚5670€voXƒÒ 2loss/SRqqt2/assignment_loss‚1.7733587935566901 2loss/SRqqt4/assignment_loss‚1.0645398363737124 /loss/FRt4/detection_loss‚1.4360533042677812 _runtime‚5670.859299858 trainer/global_step‚5299 -loss/FBt2/detection_loss‚2.64944965498788 epoch‚4 0loss/FRt4/assignment_loss‚1.3064206966332026 2loss/SRqqt1/assignment_loss‚1.3612538980586188 /loss/FRt3/detection_loss‚2.1579022236326906 0loss/SRqqt2/detection_loss‚2.241851557046175 2loss/SRqqt3/assignment_loss‚1.2545066410957864 /loss/FRt1/detection_loss‚2.4689429556222064 /loss/FBt1/detection_loss‚1.9298907319704692 _step‚435 # _timestamp‚1.78849357121914e+09 0loss/FRt1/assignment_loss‚2.8876409686130025 /loss/FBt3/detection_loss‚2.7443175464868546 /loss/FBt4/detection_loss‚0.8933909382671118 0loss/FRt2/assignment_loss‚2.9068469889461994 0loss/FRt3/assignment_loss‚2.5017780382582484 0loss/FBt2/assignment_loss‚0.7217128374447513 0loss/FBt3/assignment_loss‚0.5379601912372891 1loss/SRqqt1/detection_loss‚2.3725650634084428 0loss/SRqqt3/detection_loss‚1.320037057115273 1loss/FBt4/assignment_loss‚0.08616487404508812 .loss/FRt2/detection_loss‚2.197447538595369 1loss/SRqqt4/detection_loss‚1.0115599347012383 0loss/FBt1/assignment_loss‚0.1034129734017218 $loss/total_loss‚39.9290052437658OŽo"-„× _step‚435 # _timestamp‚1.78849357121914e+09 0loss/FRt1/assignment_loss‚2.8876409686130025 /loss/FBt3/detection_loss‚2.7443175464868546 /loss/FBt4/detection_loss‚0.8933909382671118 0loss/FRt2/assignment_loss‚2.9068469889461994 0loss/FRt3/assignment_loss‚2.50177803825éÇ•082484 0loss/FBt2/assignment_loss‚0.7217128374447513 0loss/FBt3/assignment_loss‚0.5379601912372891 1loss/SRqqt1/detection_loss‚2.3725650634084428 0loss/SRqqt3/detection_loss‚1.320037057115273 1loss/FBt4/assignment_loss‚0.08616487404508812 .loss/FRt2/detection_loss‚2.197447538595369 1loss/SRqqt4/detection_loss‚1.0115599347012383 0loss/FBt1/assignment_loss‚0.1034129734017218 $loss/total_loss‚39.9290052437658 2loss/SRqqt2/assignment_loss‚1.7733587935566901 2loss/SRqqt4/assignment_loss‚1.0645398363737124 /loss/FRt4/detection_loss‚1.4360533042677812 _runtime‚5670.859299858 trainer/global_step‚5299 -loss/FBt2/detection_loss‚2.64944965498788 epoch‚4 0loss/FRt4/assignment_loss‚1.3064206966332026 2loss/SRqqt1/assignment_loss‚1.3612538980586188 /loss/FRt3/detection_loss‚2.1579022236326906 0loss/SRqqt2/detection_loss‚2.241851557046175 2loss/SRqqt3/assignment_loss‚1.2545066410957864 /loss/FRt1/detection_loss‚2.4689429556222064 /loss/FBt1/detection_loss‚1.9298907319704692³pr0…‚(:‚ ‘þèÔë­‰¯ gpu.0.gpu‚33.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉL}冂(:ö ‘þèÔ¥ŽŠ² disk.dm-0.in‚ 22052.9921875$ memory_percent‚93.17128658294678) proc.memory.availableMB‚ 1118.81640625 disk.sda3.out‚ 169.0546875 disk.dm-1.out‚ 169.0546875( disk.nvme0c0n1.out‚140524.6083984375! disk.nvme0n1.in‚ 22052.9921875" proc.memory.rssMB‚ 2936.3046875* proc.memory.percent‚17.921781539916992 proc.cpu.threads‚17 network.sent‚ 1.2756526e+07 disk.sda.out‚ 169.0546875# disk.nvme0c0n1.in‚ 22052.9921875# disk.dm-0.out‚140524.6083984375 cpu‚25.254268378998788 network.recv‚ 1.880114e+06% disk./.usageGB‚1174.2034225463867 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚140524.6083984375) disk./.usagePercent‚78.80177690938201 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8203125 disk.sda2.in‚0þ†…‡‚(:’  þèÔ¾·­ gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.614' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.922800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03#fÔ…ˆ‚(:ú  þèÔÞ‡µ disk.sda1.out‚0' disk.nvme0n1.out‚140846.41748046875$ disk.dm-0.out‚140846.41748046875* proc.memory.percent‚17.921781539916992 proc.cpu.threads‚17) disk./.usagePercent‚78.80161920874093 disk.sda3.out‚ 169.51171875 disk.dm-1.in‚0 disk.sda.out‚ 169.51171875# disk.nvme0c0n1.in‚ 22052.9921875) disk.nvme0c0n1.out‚140846.41748046875! disk.nvme0n1.in‚ 22052.9921875$ memory_percent‚93.17135810852051 network.recv‚ 1.886554e+06 disk.sda1.in‚0 disk.sda2.out‚0 cpu‚25.253884183872696 network.sent‚ 1.2865478e+07$ disk./.usageGB‚1174.201072692871 disk.dm-1.out‚ 169.51171875 disk.drbd1260.out‚ 0.8203125 disk.sda2.in‚0 disk.dm-0.in‚ 22052.9921875( proc.memory.availableMB‚ 1118.8046875" proc.memory.rssMB‚ 2936.3046875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0"TPY‰‚(:… ¯þèÔˆ‡¹­ gpu.0.gpu‚36.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.27' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03G®‚Š‚(:÷ ¯þèÔèâõ°% proc.memory.availableMB‚ 1118.5625" proc.memory.rssMB‚ 2936.3046875$ disk./.usageGB‚1174.182014465332 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.82421875 disk.sda.in‚0) disk.nvme0c0n1.out‚141022.68310546875 disk.dm-0.in‚ 22052.9921875* proc.memory.percent‚17.921781539916992 network.recv‚ 1.890586e+06 disk.drbd1260.in‚0 disk.sda1.out‚0 cpu‚25.25348059608294 proc.cpu.threads‚17 disk.sda.out‚ 169.87890625 network.sent‚ 1.2870786e+07) disk./.usagePercent‚78.80034019509986 disk.dm-1.out‚ 169.87890625 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 22052.9921875! disk.nvme0n1.in‚ 22052.9921875 disk.sda3.out‚ 169.87890625 disk.sda1.in‚0' disk.nvme0n1.out‚141022.68310546875$ disk.dm-0.out‚141022.68310546875$ memory_percent‚93.17283630371094J~XP2‹- _wandbruntime‚5718  _runtime‚5718¼üXÇŒÁ _runtime‚5718.112946288 _step‚436 & lr-AdamW/pg1‚0.0005413876914244026 & lr-AdamW/pg2‚0.0005413876914244026 trainer/global_step‚5349 % _timestamp‚1.7884936184726188e+09©§š6ÌÆ % _timestamp‚1.7884936184726188e+09 _runtime‚5718.112946288 _step‚436 & lr-AdamW/pg1‚0.0005413876914244026 & lr-AdamW/pg2‚0.0005413876914244026 trainer/global_step‚5349´Öçó2Ž- _wandbruntime‚5718  _runtime‚5718ºt%õ'ïO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5474070852998 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83391523 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚364132508264624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53634177 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93553916 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚373673750872576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118602751 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212156665 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93553916 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83390998 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53634175 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5728518534656 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187850752 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚369607366805394 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29756823 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚414970611220480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚373673750872576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118625767 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚379167583331328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚379167583331328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211790848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚420699341545984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚420699129755136 $ _timestamp‚1.788493618473676e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5349 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212179683 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5728706385408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93553914 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚379168373979136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516ܬ ú'ôO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83390998 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53634175 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5728518534656 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187850752 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚369607366805394 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29756823 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚414970611220480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚373673750872576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118625767 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚379167583331328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚379167583331328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211790848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚420699341545984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚420699129755136 $ _timestamp‚1.788493618473676e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5349 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212179683 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5728706385408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93553914 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚379168373979136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5474070852998 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83391523 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚364132508264624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚525 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53634177 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93553916 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚373673750872576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚523 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118602751 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212156665 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93553916µžœ¡¾2‘- _wandbruntime‚5719  _runtime‚5719p›”o ’úM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93570790 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5495627865600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 _runtime‚5719.069000379 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29762014 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373741432808960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118647504 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚364198574700384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29762550 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5495627865600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118624462 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚379236288016384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415045959563264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93570788 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212195250 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118624462 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83405837 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚379237078677504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415045983503360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 _step‚438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ^JDeviceStatsMonitor.on_train_batch_end/requested_b™ŒÈòó ytes.large_pool.allocated‚364198591739744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5494855207424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5494855207424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 trainer/global_step‚5349 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373741432808960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373741450811904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93570788 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53643823 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5475842641496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5475071992876 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212218294 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212218294 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83406375 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53643825 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚420775536082432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5729576519168 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373741450811904 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚379237078677504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212195250 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93570790 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚369673646693260Û '“ÿM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5495627865600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118624462 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚379236288016384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415045959563264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93570788 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212195250 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118624462 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83405837 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚379237078677504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415045983503360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 _step‚438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚364198591739744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5494855207424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5494855207424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 trainer/global_step‚5349 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373741432808960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373741450811904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93570788 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53643823 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5475842641496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5475071992876 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212218294 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212218294 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83406375 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53643825 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚420775536082432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5729576519168 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373741450811904 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚379237078677504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212195250 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93570790 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚369673646693260 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93570790 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5495627865600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 _runtime‚5719.069000379 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29762014 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373741432808960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118647504 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚364198574700384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29762550¶ a2”- _wandbruntime‚5719  _runtime‚5719Jæ?Z•Ô 1loss/FBt2/assignment_loss‚0.30412670754032495 trainer/global_step‚5349 .loss/FRt4/detection_loss‚1.793925641477108 /loss/FBt3/detection_loss‚3.7224978655576706 _step‚439 /loss/FRt3/detection_loss‚2.0893290914595126 /loss/FBt4/detection_loss‚0.6296203870230317 0loss/FRt4/assignment_loss‚1.3746320923169455 2loss/SRqqt2/assignment_loss‚1.5406260673816388 1loss/FBt1/assignment_loss‚0.09474081842765575 1loss/FBt3/assignment_loss‚0.21662135800579563 0loss/FBt4/assignment_loss‚0.1495529755208269 1loss/SRqqt4/detection_loss‚0.9539857980532523 %loss/total_loss‚40.62953279922176 0loss/FRt1/assignment_loss‚3.5156455233693125 0loss/SRqqt2/detection_loss‚2.083143014174241 0loss/SRqqt3/detection_loss‚1.212396486521992 .loss/FBt1/detection_loss‚2.858759125073751 /loss/FRt2/assignment_loss‚2.871293041855097 .loss/FRt3/assignment_loss‚2.28331667304039 2loss/SRqqt3/assignment_loss‚1.3184597319349238 2loss/SRqqt4/assignment_loss‚1.1451629262644416 1loss/SRqqt1/detection_loss‚2.4736737585984745 % _timestamp‚1.7884936194308155e+09 _runtime‚5719.071068045 2loss/SRqqt1/assignment_loss‚1.2773144063181603 /loss/FRt1/detection_loss‚2.5875554328784345 .loss/FRt2/detection_loss‚2.301008578359604 /loss/FBt2/detection_loss‚1.8321452980691737 epoch‚4®;_–Ù epoch‚4 1loss/FBt2/assignment_loss‚0.30412670754032495 trainer/global_step‚5349 .loss/FRt4/detection_loss‚1.793925641477108 /loss/FBt3/detection_loss‚3.7224978655576706 _step‚439 /loss/FRt3/detection_loss‚2.0893290914595126 /loss/FBt4/detection_loss‚0.6296203870230317 0loss/FRt4/assignment_loss‚1.3746320923169455 2loss/SRqqt2/assignment_loss‚1.5406260673816388 1loss/FBt1/assignment_loss‚0.09474081842765575 1loss/FBt3/assignment_loss‚0.21662135800579563 0loss/FBt4/assignment_loss‚0.1495529755208269 1loss/SRqqt4/detection_loss‚0.9539857980532523 %loss/total_loss‚40.62953279922176 0loss/FRt1/assignment_loss‚3.5156455233693125 0loss/SRqqt2/detection_loss‚2.083143014174241 0loss/SRqqt3/detection_loss‚1.212396486521992 .loss/FBt1/detection_loss‚2.858759125073751 /loss/FRt2/assignment_loss‚2.871293041855097 .loss/FRt3/assignment_loss‚2.28331667304039 2loss/SRqqt3/assignment_loss‚1.3184597319349238 2loss/SRqqt4/assignment_loss‚1.1451629262644416 1loss/SRqqt1/detection_loss‚2.4736737585984745 % _timestamp‚1.7884936194308155e+09 _runtime‚5719.071068045 2loss/SRqqt1/assignment_loss‚1.2773144063181603 /loss/FRt1/detection_loss‚2.5875554328784345 .loss/FRt2/detection_loss‚2.301008578359604 /loss/FBt2/detection_loss‚1.8321452980691737·ol‘—‚(:† ¾þèÔåѲ® gpu.0.gpu‚45.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.465' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.293 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²1©V‡˜‚(:ü ¾þèÔŒù˜° disk.dm-0.in‚22180.62109375) proc.memory.availableMB‚ 1118.78515625" proc.memory.rssMB‚ 2936.3046875* proc.memory.percent‚17.921781539916992 network.recv‚ 1.894017e+06$ disk./.usageGB‚1174.182014465332 disk.sda3.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚141383.6376953125 cpu‚25.253133728896046 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.82421875 disk.sda.out‚ 170.31640625 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 170.31640625 disk.dm-1.out‚ 170.31640625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚22180.62109375# disk.dm-0.out‚141383.6376953125$ memory_percent‚93.17147731781006 network.sent‚ 1.2875782e+07) disk./.usagePercent‚78.80034019509986" disk.nvme0n1.in‚22180.62109375& disk.nvme0n1.out‚141383.6376953125r[T™‚(:… ÍþèÔ‘ßÙ¯ gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.28' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.456 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü4â‚š‚(:÷ ÍþèÔñ®Í³ disk.dm-1.in‚0 disk.sda1.out‚0' disk.nvme0n1.out‚141701.65283203125& proc.memory.availableMB‚ 1118.78125 disk.sda3.out‚ 170.6640625 disk.drbd1260.in‚0 disk.sda.out‚ 170.6640625 disk.sda1.in‚0" disk.nvme0n1.in‚22180.62109375 cpu‚25.25279437987742 proc.cpu.threads‚17( disk./.usagePercent‚78.8087428743887% disk./.usageGB‚1174.3072204589844 disk.sda3.in‚0 disk.dm-1.out‚ 170.6640625 disk.sda.in‚0$ disk.nvme0c0n1.in‚22180.62109375 disk.dm-0.in‚22180.62109375$ disk.dm-0.out‚141701.65283203125 network.sent‚ 1.298691e+07 disk.drbd1260.out‚ 0.83203125 disk.sda2.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚141701.65283203125$ memory_percent‚93.17150115966797" proc.memory.rssMB‚ 2936.3046875* proc.memory.percent‚17.921781539916992 network.recv‚ 1.901104e+068‘P›‚(:’ ÜþèÔ™«è® gpu.0.gpu‚34.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.689' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.537799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Åý;Ú{œ‚(:ð ÜþèÔ¹‹²$ disk.nvme0c0n1.in‚22180.62109375' disk.nvme0n1.out‚142362.92626953125" proc.memory.rssMB‚ 2936.3046875* proc.memory.percent‚17.921781539916992 proc.cpu.threads‚17 network.sent‚ 1.2990853e+07 disk.sda1.out‚0" disk.nvme0n1.in‚22180.62109375& proc.memory.availableMB‚ 1118.78125 cpu‚25.25241449797896 network.recv‚ 1.905422e+06) disk./.usagePercent‚78.81862553323298 disk.sda3.out‚171.1875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.83203125 disk.sda2.out‚0) disk.nvme0c0n1.out‚142362.92626953125% disk./.usageGB‚1174.4544792175293 disk.sda3.in‚0 disk.dm-1.out‚171.1875 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚22180.62109375$ disk.dm-0.out‚142362.92626953125$ memory_percent‚93.17150115966797 disk.sda.out‚171.1875 disk.sda2.in‚0@\s{2- _wandbruntime‚5768  _runtime‚5768#)¡•ÇžÁ _runtime‚5768.464825175 _step‚440 & lr-AdamW/pg1‚0.0005360526844163861 & lr-AdamW/pg2‚0.0005360526844163861 trainer/global_step‚5399 % _timestamp‚1.7884936688243747e+09s&…“ÌŸÆ & lr-AdamW/pg1‚0.0005360526844163861 & lr-AdamW/pg2‚0.0005360526844163861 trainer/global_step‚5399 % _timestamp‚1.7884936688243747e+09 _runtime‚5768.464825175 _step‚440¸ÊŽw2 - _wandbruntime‚5768  _runtime‚5768ñJ8 õ'¡ïO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _step‚441 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94397616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚367435847091984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382602831997952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5781099480576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84139301 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377058747838976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5544874806784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54116577 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377058729836032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119691653 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5544874806784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5524912086518 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382603622645760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377058747838976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5780903241216 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚367435830052624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382602831997952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚418738028359680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54116575 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 _runtime‚ 5768.46603777 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚372959971490730 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30022724 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119714669 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119691653 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377058747838976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5780903241216 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚367435830052624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382602831997952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚418738028359680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.frU¿û‚ eed‚54116575 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 _runtime‚ 5768.46603777 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚372959971490730 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30022724 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119714669 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119691653 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382602831997952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5781099480576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84139301 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377058747838976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5544874806784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54116577 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377058729836032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119691653 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5544874806784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0¹ ¶w/2£- _wandbruntime‚5769  _runtime‚5769•¼ì'¤úM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119736406 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214150896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94414488 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119736406 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚373027826754348 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214150896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 trainer/global_step‚5399 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84153733 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5525142577984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5545106907648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5781948097024 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54126225 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94414488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424595547062784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884936696845279e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367501896488384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5545879565824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377126429775360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373027039066368 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367501913527744 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94414490 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119713364 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382672327344128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30027510 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382671536683008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3771264297753604~@5'¥ÿM EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367501913527744 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94414490 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119713364 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382672327344128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30027510 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382671536683008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚377126429775360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119736406 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214150896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94414488 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119736406 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚373027826754348 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214150896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 trainer/global_step‚5399 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84153733 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5525142577984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5545106907648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5781948097024 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54126225 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94414488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424595547062784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884936696845279e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367501896488384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5545879565824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377126429775360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373027039066368ºDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386038080664576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95241314 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84886267 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30287292 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5574212023214 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95241316 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95241314 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚380443726802432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5594353862144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380443744805376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120780555 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54598975 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5594353862144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376313363863838 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84886809 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5833543586304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5833347346944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370739151840624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30287832 % _timestamp‚1.7884937182004282e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386038871312384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 _step‚445 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386038871312384 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚422505469438976 trainer/global_step‚5449 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚376314151551610 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95241316 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380443726802432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216044887 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360YAðû'´õO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386038871312384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 _step‚445 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386038871312384 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚422505469438976 trainer/global_step‚5449 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚376314151551610 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95241316 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380443726802432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216044887 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5595126507008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5574982671626 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386038080664576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95241314 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84886267 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30287292 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5574212023214 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95241316 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95241314 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚380443726802432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5594353862144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380443744805376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120780555 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54598975 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5594353862144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376313363863838 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84886809 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5833543586304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5833347346944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370739151840624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30287832 % _timestamp‚1.7884937182004282e+09 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884937191007605e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95258188 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5834590076416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120825308 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216060454 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216083498 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5575213163092 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚5818.742018181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚370805218276384 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5596131266048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386107576010752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95258190 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30293114 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5596131266048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428415407858176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95258190 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380511426741760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5595358607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84901739 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216083498 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54608623 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5575983811712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386107576010752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚380511426741760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120825308 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5595358607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84901200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚422580793841664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54608625 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884937191007605e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95258188 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5834590076416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120825308 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216060454 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216083498 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5575213163092 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚5818.742018181 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚370805218276384 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5596131266048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386107576010752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95258190 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30293114 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5596131266048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428415407858176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95258190 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380511426741760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5595358607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84901739 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216083498 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54608623 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5575983811712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386107576010752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚380511426741760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120825308 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5595358607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84901200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚422580793841664¾…4¶¿2¸- _wandbruntime‚5818  _runtime‚5818*){Z¹Ô .loss/FRt2/detection_loss‚2.645360086645399 0loss/FRt1/assignment_loss‚2.6082214477815127 0loss/FRt2/assignment_loss‚3.2124118770871846 2loss/SRqqt3/assignment_loss‚1.6030813331222709 0loss/FBt4/assignment_loss‚0.1557827027348297 0loss/SRqqt2/detection_loss‚2.218552080147407 epoch‚4 %loss/total_loss‚41.11708662218959 2loss/SRqqt2/assignment_loss‚1.5143624059855938 1loss/FBt1/assignment_loss‚0.08384088535482685 1loss/FBt3/assignment_loss‚0.12654444776126184 _runtime‚5818.743967844 1loss/SRqqt3/detection_loss‚1.4094991483451689 _step‚447 /loss/FRt3/assignment_loss‚2.262537489997016 /loss/FBt2/detection_loss‚2.7762729078531265 0loss/FRt4/assignment_loss‚1.4818706134190927 1loss/SRqqt1/assignment_loss‚1.291765377049645 2loss/SRqqt4/assignment_loss‚1.1001606868383915 1loss/FBt2/assignment_loss‚0.14881836269699988 -loss/FRt3/detection_loss‚2.20766408542792 % _timestamp‚1.7884937191037772e+09 /loss/FRt1/detection_loss‚2.5852995554083273 1loss/SRqqt4/detection_loss‚0.8262089829107648 .loss/FBt1/detection_loss‚3.210096458594004 /loss/FBt3/detection_loss‚3.1150185267130532 trainer/global_step‚5449 /loss/FRt4/detection_loss‚1.5393720731998866 0loss/SRqqt1/detection_loss‚2.446503621339798 /loss/FBt4/detection_loss‚0.5478414657761107æÊæ_ºÙ /loss/FRt3/assignment_loss‚2.262537489997016 /loss/FBt2/detection_loss‚2.7762729078531265 0loss/FRt4/assignment_loss‚1.4818706134190927 1loss/SRqqt1/assignment_loss‚1.291765377049645 2loss/SRqqt4/assignment_loss‚1.1001606868383915 1loss/FBt2/assignment_loss‚0.14881836269699988 -loss/FRt3/detection_loss‚2.20766408542792 % _timestamp‚1.7884937191037772e+09 /loss/FRt1/detection_loss‚2.5852995554083273 1loss/SRqqt4/detection_loss‚0.8262089829107648 .loss/FBt1/detection_loss‚3.210096458594004 /loss/FBt3/detection_loss‚3.1150185267130532 trainer/global_step‚5449 /loss/FRt4/detection_loss‚1.5393720731998866 0loss/SRqqt1/detection_loss‚2.446503621339798 /loss/FBt4/detection_loss‚0.5478414657761107 .loss/FRt2/detection_loss‚2.645360086645399 0loss/FRt1/assignment_loss‚2.6082214477815127 0loss/FRt2/assignment_loss‚3.2124118770871846 2loss/SRqqt3/assignment_loss‚1.6030813331222709 0loss/FBt4/assignment_loss‚0.1557827027348297 0loss/SRqqt2/detection_loss‚2.218552080147407 epoch‚4 %loss/total_loss‚41.11708662218959 2loss/SRqqt2/assignment_loss‚1.5143624059855938 1loss/FBt1/assignment_loss‚0.08384088535482685 1loss/FBt3/assignment_loss‚0.12654444776126184 _runtime‚5818.743967844 1loss/SRqqt3/detection_loss‚1.4094991483451689 _step‚447¿6™ûT»‚(:’ ˜ÿèÔÞÜ‘¯ gpu.0.gpu‚10.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.791000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø(ˆ†¼‚(:û ˜ÿèÔö˜³ disk.sda1.in‚0' disk.nvme0n1.out‚143667.50341796875$ memory_percent‚93.16954612731934* proc.memory.percent‚17.921781539916992% disk./.usageGB‚1174.6809043884277 disk.drbd1260.out‚ 0.83984375$ disk.nvme0c0n1.in‚22317.96484375) disk.nvme0c0n1.out‚143667.50341796875" disk.nvme0n1.in‚22317.96484375 disk.dm-0.in‚22317.96484375" proc.memory.rssMB‚ 2936.3046875 network.sent‚ 1.3116498e+07 disk.dm-1.out‚ 172.8359375 disk.sda.out‚ 172.8359375 disk.sda1.out‚0 disk.sda2.out‚0$ disk.dm-0.out‚143667.50341796875 proc.cpu.threads‚17 network.recv‚ 1.92393e+06 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 1119.1015625 cpu‚25.250856589250272) disk./.usagePercent‚78.83382111643535 disk.sda3.out‚ 172.8359375 disk.dm-1.in‚0 disk.sda.in‚08"Y”‘½‚(:† §ÿèÔ˜™ß­ gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.015' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.203 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,Ã{¾‚(:ð §ÿèÔóÐï¹ disk.sda3.in‚0% proc.memory.availableMB‚ 1119.0625* proc.memory.percent‚17.921781539916992 network.sent‚ 1.3226116e+07$ disk.nvme0c0n1.in‚22317.96484375 disk.dm-0.in‚22317.96484375 disk.dm-1.in‚0 disk.dm-1.out‚ 173.21484375 disk.drbd1260.in‚0 disk.sda.out‚ 173.21484375$ memory_percent‚93.16978454589844 disk.sda2.out‚0$ disk.nvme0n1.out‚144136.73828125 disk.sda3.out‚ 173.21484375 disk.drbd1260.out‚0.84375 disk.sda.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2936.3046875 proc.cpu.threads‚17) disk./.usagePercent‚78.86737525592561% disk./.usageGB‚1175.1808853149414 disk.sda2.in‚0& disk.nvme0c0n1.out‚144136.73828125 disk.sda1.out‚0 cpu‚25.250500833650097 network.recv‚ 1.930095e+06" disk.nvme0n1.in‚22317.96484375! disk.dm-0.out‚144136.73828125[ma<’¿‚(:‡ ¶ÿèÔúá‚­ gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.779' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.9558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l6ÒoÀ‚(:ä ¶ÿèÔ˃±% disk.nvme0c0n1.out‚144563.5390625 disk.dm-0.out‚144563.5390625% proc.memory.availableMB‚ 1118.8125 cpu‚25.2501545771889 network.recv‚ 1.93332e+06% disk./.usageGB‚1175.1515808105469 disk.dm-1.in‚0$ memory_percent‚93.17131042480469" proc.memory.rssMB‚ 2936.3046875 disk.dm-1.out‚ 173.75390625 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 22441.296875 disk.nvme0n1.in‚ 22441.296875 disk.sda3.in‚0 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚0.84375 disk.sda.in‚0 disk.sda1.in‚0# disk.nvme0n1.out‚144563.5390625 disk.dm-0.in‚ 22441.296875 disk.sda3.out‚ 173.75390625* proc.memory.percent‚17.921781539916992 network.sent‚ 1.3229236e+07) disk./.usagePercent‚78.86540860604758 disk.sda.out‚ 173.75390625 disk.sda1.out‚0ô ÌÁ‚(:’ ÅÿèÔ÷Ú® gpu.0.gpu‚37.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.118' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.023599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y(¸¹‚‚(:÷ ÅÿèÔ‹Ÿê± disk.sda1.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2936.3046875 proc.cpu.threads‚17) disk./.usagePercent‚78.86541244616059# disk.dm-0.out‚144827.4755859375 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.84765625# memory_percent‚93.1682825088501) proc.memory.availableMB‚ 1119.30859375 cpu‚25.24981036871996 network.sent‚ 1.323547e+07 network.recv‚ 1.938338e+06# disk.nvme0c0n1.in‚ 22443.8828125( disk.nvme0c0n1.out‚144827.4755859375! disk.nvme0n1.in‚ 22443.8828125 disk.sda.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.921781539916992% disk./.usageGB‚1175.1516380310059& disk.nvme0n1.out‚144827.4755859375 disk.dm-0.in‚ 22443.8828125 disk.sda3.out‚ 174.25390625 disk.dm-1.out‚ 174.25390625 disk.sda.out‚ 174.25390625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚01J“ã2Ã- _wandbruntime‚5867  _runtime‚5867ÀS­¯ÆÄÀ & lr-AdamW/pg1‚0.0005252963036589851 & lr-AdamW/pg2‚0.0005252963036589851 trainer/global_step‚5499 % _timestamp‚1.7884937679124768e+09 _runtime‚ 5867.55292952 _step‚448æþ~8ËÅÅ & lr-AdamW/pg2‚0.0005252963036589851 trainer/global_step‚5499 % _timestamp‚1.7884937679124768e+09 _runtime‚ 5867.55292952 _step‚448 & lr-AdamW/pg1‚0.0005252963036589851À­Ó8<2Æ- _wandbruntime‚5867  _runtime‚5867[P†€õ'ÇïO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217954471 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121869457 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5644605562368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121892473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 $ _timestamp‚1.788493767913831e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5645378207232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217977489 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚374042490667984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30552685 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5885678993408 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379667543924718 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426272886578176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚432158757616640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚383828741771776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚383828723768832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389474119979008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5644605562368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379666756236946 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85634601 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5625053256734 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389474119979008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55081377 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚5867.554437151 _step‚449 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96085016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389473329331200 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85634060 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30553224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96085014 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚383828741771776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96085016 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121892473 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5645378207232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚5499 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚389473329331200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125ˆvPM² ÈôO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5645378207232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217977489 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚374042490667984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30552685 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5885678993408 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379667543924718 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426272886578176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚432158757616640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚383828741771776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚383828723768832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389474119979008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDÕvôHeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5644605562368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379666756236946 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85634601 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5625053256734 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389474119979008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55081377 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚5867.554437151 _step‚449 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96085016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389473329331200 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85634060 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30553224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96085014 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚383828741771776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96085016 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121892473 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5645378207232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚5499 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚389473329331200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217954471 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121869457 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5644605562368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121892473 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 $ _timestamp‚1.788493767913831e+09Áø•k2É- _wandbruntime‚5868  _runtime‚5868ÐßÇ°ÿ&ÊùM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217993056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185740288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389542824677376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5646382966272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96101890 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121891168 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218016100 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚432234953238528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚374108557103744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚383896423708160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96101890 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚383896423708160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚450 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5646382966272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30558447 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚432235162918912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85648937 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚389542824677376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121914210 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217993056 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379733823812584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5886742257664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚374108540064384 $ _timestamp‚1.788493768814857e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5625283748200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96101888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121914210 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218016100 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚426348210980864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379734611500564 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55091025 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85649472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5645610308096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30557914 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489[ˆ±'ËþM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚389542824677376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121914210 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217993056 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379733823812584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5886742257664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚374108540064384 $ _timestamp‚1.788493768814857e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5625283748200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96101888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121914210 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218016100 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚426348210980864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379734611500564 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55091025 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85649472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5645610308096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30557914 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217993056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185740288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389542824677376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5646382966272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96101890 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121891168 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218016100 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚432234953238528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚374108557103744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚383896423708160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96101890 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚383896423708160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚450 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5646382966272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30558447 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚432235162918912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85648937 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624Âd‹>´2Ì- _wandbruntime‚5868  _runtime‚5868ßmöYÍÓ 0loss/FBt1/assignment_loss‚0.3455155166879599 epoch‚4 .loss/FBt2/detection_loss‚2.067628745849316 /loss/FBt3/detection_loss‚2.7329287081956863 trainer/global_step‚5499 % _timestamp‚1.7884937688174543e+09 /loss/FRt2/assignment_loss‚3.589399723445668 /loss/FRt4/detection_loss‚1.5837279979884624 /loss/FRt3/assignment_loss‚2.060971647500992 1loss/FBt3/assignment_loss‚0.19029955560654344 1loss/SRqqt4/detection_loss‚0.9533896336000259 .loss/FRt2/detection_loss‚2.530706786057943 1loss/SRqqt1/detection_loss‚2.3277089534436954 2loss/SRqqt4/assignment_loss‚0.8792064729184933 .loss/FRt1/detection_loss‚2.611398361849062 .loss/FRt3/detection_loss‚2.175319143486294 2loss/SRqqt1/assignment_loss‚1.5061267908005154 1loss/SRqqt3/detection_loss‚1.1448724815208051 /loss/FBt1/detection_loss‚2.8268664711051517 /loss/FBt4/detection_loss‚0.7313901121035601 _runtime‚5868.457603817 _step‚451 2loss/SRqqt2/assignment_loss‚1.4820140382165419 2loss/SRqqt3/assignment_loss‚1.6255436437721882 0loss/FBt2/assignment_loss‚0.3242155509606754 1loss/FBt4/assignment_loss‚0.06820397631405338 0loss/SRqqt2/detection_loss‚2.327525512260549 %loss/total_loss‚40.42427971557768 /loss/FRt1/assignment_loss‚3.088418241703149 0loss/FRt4/assignment_loss‚1.2509016501903534Cë‹^ÎØ .loss/FRt3/detection_loss‚2.175319143486294 2loss/SRqqt1/assignment_loss‚1.5061267908005154 1loss/SRqqt3/detection_loss‚1.1448724815208051 /loss/FBt1/detection_loss‚2.8268664711051517 /loss/FBt4/detection_loss‚0.7313901121035601 _runtime‚5868.457603817 _step‚451 2loss/SRqqt2/assignment_loss‚1.4820140382165419 2loss/SRqqt3/assignment_loss‚1.6255436437721882 0loss/FBt2/assignment_loss‚0.3242155509606754 1loss/FBt4/assignment_loss‚0.06820397631405338 0loss/SRqqt2/detection_loss‚2.327525512260549 %loss/total_loss‚40.42427971557768 /loss/FRt1/assignment_loss‚3.088418241703149 0loss/FRt4/assignment_loss‚1.2509016501903534 0loss/FBt1/assignment_loss‚0.3455155166879599 epoch‚4 .loss/FBt2/detection_loss‚2.067628745849316 /loss/FBt3/detection_loss‚2.7329287081956863 trainer/global_step‚5499 % _timestamp‚1.7884937688174543e+09 /loss/FRt2/assignment_loss‚3.589399723445668 /loss/FRt4/detection_loss‚1.5837279979884624 /loss/FRt3/assignment_loss‚2.060971647500992 1loss/FBt3/assignment_loss‚0.19029955560654344 1loss/SRqqt4/detection_loss‚0.9533896336000259 .loss/FRt2/detection_loss‚2.530706786057943 1loss/SRqqt1/detection_loss‚2.3277089534436954 2loss/SRqqt4/assignment_loss‚0.8792064729184933 .loss/FRt1/detection_loss‚2.611398361849062ød]T„Ï‚(:ù ÔÿèÔÓÔ±°) disk./.usagePercent‚78.86383748780995# disk.nvme0c0n1.in‚ 22443.8828125! disk.nvme0n1.in‚ 22443.8828125 disk.dm-0.in‚ 22443.8828125 disk.sda3.out‚ 174.62109375 disk.drbd1260.out‚ 0.84765625 network.recv‚ 1.94247e+06( disk.nvme0c0n1.out‚145098.1787109375# disk.dm-0.out‚145098.1787109375 disk.sda3.in‚0 disk.sda.out‚ 174.62109375" proc.memory.rssMB‚ 2936.3046875 cpu‚25.249404334829347% disk./.usageGB‚1175.1281700134277 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚145098.1787109375 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚93.17305088043213) proc.memory.availableMB‚ 1118.52734375* proc.memory.percent‚17.921781539916992 proc.cpu.threads‚17 disk.dm-1.out‚ 174.62109375 network.sent‚ 1.3241363e+07ü–#‘’Ђ(:‡ ÔÿèÔªáÞ¯ gpu.0.gpu‚48.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.573' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5146 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.]šý’Ñ‚(:‡ ãÿèÔ£»•­ gpu.0.gpu‚45.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚167.377' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò®¹º{Ò‚(:ð ãÿèÔÖÀè®( disk.nvme0c0n1.out‚145663.6904296875 disk.sda3.out‚ 174.9765625 disk.drbd1260.out‚ 0.8515625$ memory_percent‚93.17007064819336 cpu‚25.2490427159629 proc.cpu.threads‚17 network.recv‚ 1.948326e+06# disk.nvme0c0n1.in‚ 22443.8828125! disk.nvme0n1.in‚ 22443.8828125 disk.dm-0.in‚ 22443.8828125# disk.dm-0.out‚145663.6904296875 disk.sda.in‚0 network.sent‚ 1.3349443e+07& disk.nvme0n1.out‚145663.6904296875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 174.9765625 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2936.3046875 disk.dm-1.out‚ 174.9765625) disk./.usagePercent‚78.86536150066128# disk./.usageGB‚1175.15087890625 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1119.015625* proc.memory.percent‚17.921781539916992È8/6#Ó‚(:† òÿèÔ³¶È® gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚170.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.023 gpu.0.smClæ=cônock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z¨Øy€Ô‚(:õ òÿèÔÌÛ·± cpu‚25.2487718904909 proc.cpu.threads‚17 network.sent‚ 1.3354803e+07) disk./.usagePercent‚78.86378654231063% disk./.usageGB‚1175.1274108886719 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.8515625 disk.sda1.out‚0( disk.nvme0c0n1.out‚146102.5224609375! disk.nvme0n1.in‚ 22443.8828125 network.recv‚ 1.952071e+06 disk.sda3.out‚ 175.36328125 disk.sda.in‚0 disk.sda.out‚ 175.36328125 disk.sda2.out‚0 disk.dm-0.in‚ 22443.8828125# disk.dm-0.out‚146102.5224609375* proc.memory.percent‚17.921781539916992 disk.sda3.in‚0 disk.dm-1.out‚ 175.36328125 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 22443.8828125' proc.memory.availableMB‚ 1118.765625" proc.memory.rssMB‚ 2936.3046875 disk.drbd1260.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚146102.5224609375$ memory_percent‚93.17159652709961ÌfRÞ2Õ- _wandbruntime‚5917  _runtime‚5917¤ˆóÜÇÖÁ _step‚452 & lr-AdamW/pg1‚0.0005198771715232283 & lr-AdamW/pg2‚0.0005198771715232283 trainer/global_step‚5549 % _timestamp‚1.7884938181152084e+09 _runtime‚5917.755659517YPblÌ×Æ & lr-AdamW/pg2‚0.0005198771715232283 trainer/global_step‚5549 % _timestamp‚1.7884938181152084e+09 _runtime‚5917.755659517 _step‚452 & lr-AdamW/pg1‚0.0005198771715232283ÄN’ 2Ø- _wandbruntime‚5917  _runtime‚5917Åê#¬õ'ÙïO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122981375 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430040303717376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5675123841842 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96928716 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392908577997824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5695629907456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 _runtime‚ 5917.75782659 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96928716 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122958359 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5674353193430 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435978187623424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚377345795416624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚377345812455984 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122958359 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚453 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122981375 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5938118765568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576’ƒWú'ÚôO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122981375 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5938118765568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122981375 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430040303717376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5675123841842 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96928716 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392908577997824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5695629907456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 _runtime‚ 5917.75782659 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96928716 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122958359 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5674353193430 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435978187623424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚377345795416624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚377345812455984 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122958359 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚453ÅŠˆ§2Û- _wandbruntime‚5918  _runtime‚5918/fŒü&ÜöM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5695862008320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚553 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5695862008320 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96945588 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219925658 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123003112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122980070 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5938952798208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚377411878891744 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 trainer/global_step‚5549 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30823558 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚392977282682880 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219948702 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96945588 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚436054580918272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387281420674560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 " _timestamp‚1.7884938191748e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 219925658 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚377411861852384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430115628120064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387281438677504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122980070 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚392978073344000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚551 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5696634666496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86396981 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86397534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30824109 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55573425 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5675354333308 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219948702 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5939153218560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚ 5918.81657234 _step‚454 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387281420674560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚436054805278720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0dUO±ÝûM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚436054805278720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5695862008320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 B9DeviceStatsMonitor.on_train_batch_en眥P$d/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚553 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5695862008320 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96945588 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219925658 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123003112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122980070 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5938952798208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚377411878891744 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 trainer/global_step‚5549 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30823558 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚392977282682880 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219948702 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96945588 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚436054580918272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387281420674560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 " _timestamp‚1.7884938191748e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 219925658 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚377411861852384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430115628120064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387281438677504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122980070 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚392978073344000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚551 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5696634666496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86396981 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86397534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30824109 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55573425 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5675354333308 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219948702 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5939153218560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚ 5918.81657234 _step‚454 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387281420674560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 PgD{æ‚(:𠟀éÔÒáþ¯ disk.sda.in‚0 disk.sda.out‚ 176.73046875 disk.nvme0n1.in‚ 22589.59375 disk.drbd1260.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 22589.59375 disk.dm-0.in‚ 22589.59375" proc.memory.rssMB‚ 2936.3046875 proc.cpu.threads‚17 disk.drbd1260.out‚0.859375 disk.sda1.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚147119.5107421875# disk.dm-0.out‚147119.5107421875 cpu‚25.24767513610396 network.recv‚ 1.967206e+06) disk./.usagePercent‚78.86536892487977 disk.sda3.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚147119.5107421875$ memory_percent‚93.17221641540527( proc.memory.availableMB‚ 1118.6640625* proc.memory.percent‚17.921781539916992 disk.sda1.out‚0 network.sent‚ 1.3475006e+07% disk./.usageGB‚1175.1509895324707 disk.sda3.out‚ 176.73046875 disk.dm-1.out‚ 176.73046875òL„O2ç- _wandbruntime‚5967  _runtime‚5967iéJÉÇèÁ _step‚456 & lr-AdamW/pg1‚0.0005144322487980962 & lr-AdamW/pg2‚0.0005144322487980962 trainer/global_step‚5599 % _timestamp‚1.7884938678622723e+09 _runtime‚5967.502579509Ï ×ÌéÆ & lr-AdamW/pg1‚0.0005144322487980962 & lr-AdamW/pg2‚0.0005144322487980962 trainer/global_step‚5599 % _timestamp‚1.7884938678622723e+09 _runtime‚5967.502579509 _step‚456Èp¸È12ê- _wandbruntime‚5967  _runtime‚5967„mŒõ'ëïO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56046175 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124047261 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚386374328670934 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚390598735704576 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 221842693 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚439798177125888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 _runtime‚5967.503967173 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5990257932800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚396344617312256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31083391 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚439797954849280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5725194426950 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97772414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚396343826664448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5990456269312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788493867863352e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5745108962816 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚396343826664448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390598717701632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5745881607680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 trainer/global_step‚5599 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 221842693 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚439798177125888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 _runtime‚5967.503967173 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5990257932800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚396344617312256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31083391 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚439797954849280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5725194426950 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97772414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚396343826664448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5990456269312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788493867863352e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5745108962816 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚396343826664448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390598717701632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5743ˆÎ5881607680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 trainer/global_step‚5599 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚386374328670934 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚390598735704576 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221881304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87145013 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5991314762240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚458 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124092014 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97789288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚390666435643904 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5991506793984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚5599 % _timestamp‚1.7884938687242403e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5746886366720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97789290 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚386440608558800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5726195567036 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 221858260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚439874360021504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5746113708544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5746113708544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚396412531349504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56055823 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31088657 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380715183640384 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚396413322010624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380715200679744 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124068972 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87144480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096é›»'ïÿM trainer/global_step‚5599 % _timestamp‚1.7884938687242403e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5746886366720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97789290 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚386440608558800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5726195567036 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 221858260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚439874360021504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5746113708544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5746113708544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚396412531349504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56055823 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31088657 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380715183640384 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚396413322010624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380715200679744 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124068972 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87144480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221881304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87145013 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5991314762240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚458 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124092014 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97789288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚390666435643904 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5991506793984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328ÊÄDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437575137995776 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚389727721044042 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399779075331072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443618031474688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437575114055680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚6017.915919173 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98616116 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5649 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 223752277 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98616116 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125159179 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5796133307904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5774494363646 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚567 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399779865978880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399779075331072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125159179 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223775295 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6042893478912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5795360663040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125136163 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5796133307904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31348205 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚443617798712320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5775265012058 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56528577 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516ˆÉ‡®ú'€ôO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98616114 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125136163 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87876780 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98616114 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6042684656640 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚383952438992624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208822272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437575114055680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚6017.915919173 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98616116 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5649 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 223752277 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98616116 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125159179 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5796133307904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5774494363646 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚567 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399779865978880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399779075331072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125159179 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223775295 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6042893478912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5795360663040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125136163 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5796133307904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31348205 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚443617798712320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5775265012058 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56528577 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚393983732670976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚393983714668032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31348772 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232762368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚393983732670976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚383952456031984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87877349 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚569 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56528575 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚393983714668032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 $ _timestamp‚1.788493918275157e+09 _step‚461 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5795360663040 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚389726933356270 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437575137995776 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚389727721044042 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399779075331072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443618031474688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚389794788619888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223813906 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394051414607360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56538223 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437650462398464 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98632990 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87891760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚394051432610304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384018505428384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223813906 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 trainer/global_step‚5649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223790862 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5797138066944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5796365408768 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98632988 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚546 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399848570677248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31354083 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223790862 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443694206796800 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384018522467744Ã}±+ƒÿM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31353537 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚462 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125180916 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5797138066944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚389794000931908 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚389794788619888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223813906 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394051414607360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56538223 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437650462398464 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98632990 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87891760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚394051432610304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384018505428384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223813906 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 trainer/global_step‚5649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223790862 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5797138066944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5796365408768 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98632988 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚546 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399848570677248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31354083 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223790862 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443694206796800 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384018522467744Îe€¢Ò2„- _wandbruntime‚6018  _runtime‚6018 …(Z…Ô 0loss/FRt3/assignment_loss‚1.9862944002334888 trainer/global_step‚5649 _runtime‚6018.980222506 -loss/FRt1/detection_loss‚2.31206303326095 /loss/FRt2/detection_loss‚2.7750430264820656 /loss/FRt3/detection_loss‚1.8657340456086855 1loss/SRqqt3/detection_loss‚1.3624854049496893 1loss/SRqqt4/detection_loss‚0.9717039874557293 %loss/total_loss‚41.29920499784102 0loss/FRt2/assignment_loss‚3.1727286875247955 2loss/SRqqt2/assignment_loss‚1.3773789972066879 1loss/SRqqt3/assignment_loss‚1.202852213235599 /loss/FRt4/detection_loss‚1.7641063580370469 % _timestamp‚1.7884939193400254e+09 2loss/SRqqt1/assignment_loss‚1.7602715976536274 1loss/FBt4/assignment_loss‚0.14183241978564584 .loss/FBt1/detection_loss‚2.616260674264696 .loss/FBt3/detection_loss‚3.307593206564585 /loss/FRt1/assignment_loss‚3.286219819289882 0loss/FRt4/assignment_loss‚1.5684489335702814 0loss/FBt1/assignment_loss‚0.2760350739707873 2loss/FBt2/assignment_loss‚0.008465350274895577 0loss/FBt3/assignment_loss‚0.4029711907860474 0loss/SRqqt2/detection_loss‚2.406769984960556 .loss/FBt2/detection_loss‚2.740202479892307 /loss/FBt4/detection_loss‚0.6049029682871214 2loss/SRqqt4/assignment_loss‚1.0245447549263411 1loss/SRqqt1/detection_loss‚2.3642963896195095 epoch‚4 _step‚463Ñ·zl_†Ù 2loss/SRqqt1/assignment_loss‚1.7602715976536274 1loss/FBt4/assignment_loss‚0.14183241978564584 .loss/FBt1/detection_loss‚2.616260674264696 .loss/FBt3/detection_loss‚3.307593206564585 /loss/FRt1/assignment_loss‚3.286219819289882 0loss/FRt4/assignment_loss‚1.5684489335702814 0loss/FBt1/assignment_loss‚0.2760350739707873 2loss/FBt2/assignment_loss‚0.008465350274895577 0loss/FBt3/assignment_loss‚0.4029711907860474 0loss/SRqqt2/detection_loss‚2.406769984960556 .loss/FBt2/detection_loss‚2.740202479892307 /loss/FBt4/detection_loss‚0.6049029682871214 2loss/SRqqt4/assignment_loss‚1.0245447549263411 1loss/SRqqt1/detection_loss‚2.3642963896195095 epoch‚4 _step‚463 0loss/FRt3/assignment_loss‚1.9862944002334888 trainer/global_step‚5649 _runtime‚6018.980222506 -loss/FRt1/detection_loss‚2.31206303326095 /loss/FRt2/detection_loss‚2.7750430264820656 /loss/FRt3/detection_loss‚1.8657340456086855 1loss/SRqqt3/detection_loss‚1.3624854049496893 1loss/SRqqt4/detection_loss‚0.9717039874557293 %loss/total_loss‚41.29920499784102 0loss/FRt2/assignment_loss‚3.1727286875247955 2loss/SRqqt2/assignment_loss‚1.3773789972066879 1loss/SRqqt3/assignment_loss‚1.202852213235599 /loss/FRt4/detection_loss‚1.7641063580370469 % _timestamp‚1.7884939193400254e+09ÏØ’Z‡‚(:’ ê€éÔ–‚® gpu.0.gpu‚48.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.497' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.299400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ¥êsˆ‚(:è ê€éÔþþè® disk.drbd1260.in‚0 disk.sda.out‚ 178.67578125 disk.sda2.out‚0 disk.nvme0n1.in‚22721.75% proc.memory.availableMB‚ 1118.5625" proc.memory.rssMB‚ 2936.3046875 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.989559e+06% disk./.usageGB‚1173.9911422729492 disk.sda3.out‚ 178.67578125 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.8671875 disk.nvme0c0n1.in‚22721.75 disk.dm-0.in‚22721.75* proc.memory.percent‚17.921781539916992 cpu‚25.24583153528981 proc.cpu.threads‚17 network.sent‚ 1.3605658e+07) disk./.usagePercent‚78.78753060211662 disk.sda.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚148925.43798828125' disk.nvme0n1.out‚148925.43798828125$ disk.dm-0.out‚148925.43798828125$ memory_percent‚93.17283630371094 disk.sda3.in‚0 disk.dm-1.out‚ 178.67578125Q²Õœ‰‚(:‘ ù€éÔÛÌ›­ gpu.0.gpu‚39.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.181' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚33.43620000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qv¾„Š‚(:ù ù€éÔ·ÍÞ® disk.sda.out‚ 179.234375 disk.sda2.in‚0* proc.memory.percent‚17.921781539916992 proc.cpu.threads‚17 network.recv‚ 1.997116e+06% disk./.usageGB‚1173.9902420043945 disk.dm-0.in‚22851.49609375 disk.sda3.in‚0 disk.dm-1.out‚ 179.234375 cpu‚25.245515952668388 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚149118.60009765625' disk.nvme0n1.out‚149118.60009765625) proc.memory.availableMB‚ 1118.56640625 disk.drbd1260.out‚ 0.87109375 network.sent‚ 1.371536e+07) disk./.usagePercent‚78.78747018433853 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚22851.49609375" disk.nvme0n1.in‚22851.49609375$ memory_percent‚93.17281246185303" proc.memory.rssMB‚ 2936.3046875$ disk.dm-0.out‚149118.60009765625 disk.sda3.out‚ 179.234375nϪq‹‚(:’ ˆéÔþ‹ñ¯ gpu.0.gpu‚38.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚125.439' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.087799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ï^ñ‰Œ‚(:þ ˆéÔ˜Àë² disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚22851.49609375$ memory_percent‚93.17286014556885 network.sent‚ 1.3721153e+07 disk.dm-1.out‚ 179.76953125 disk.sda1.in‚0$ disk.nvme0c0n1.in‚22851.49609375 disk.dm-0.in‚22851.49609375" proc.memory.rssMB‚ 2936.3046875) disk./.usagePercent‚78.71346045424994 disk.dm-1.in‚0 disk.sda.out‚ 179.76953125$ disk.dm-0.out‚149617.61962890625 cpu‚25.24512547278338 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd1260.out‚ 0.87109375 disk.sda.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚149617.61962890625' disk.nvme0n1.out‚149617.61962890625) proc.memory.availableMB‚ 1118.55859375* proc.memory.percent‚17.921781539916992 network.recv‚ 2.00138e+06% disk./.usageGB‚1172.8874435424805 disk.sda3.out‚ 179.76953125 disk.drbd1260.in‚0—´û$2- _wandbruntime‚6066  _runtime‚6066´~¯ÇŽÁ & lr-AdamW/pg1‚0.0005034695755728925 & lr-AdamW/pg2‚0.0005034695755728925 trainer/global_step‚5699 % _timestamp‚1.7884939670766306e+09 _runtime‚6066.717138647 _step‚464¹X€´ÌÆ trainer/global_step‚5699 % _timestamp‚1.7884939670766306e+09 _runtime‚6066.717138647 _step‚464 & lr-AdamW/pg1‚0.0005034695755728925 & lr-AdamW/pg2‚0.0005034695755728925ÐhœÂ2- _wandbruntime‚6066  _runtime‚6066“2’ƒö'‘ðO BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126225065 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99459816 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126248081 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441342531194880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403215114645504 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚554 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126248081 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387255760780624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5825335597166 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6095366928896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚393081113417150 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31614764 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚441342555134976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403214323997696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397368711634432 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99459816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88626295 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚393080325729378 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚387255777819984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99459814 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99459814 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5846385008128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225684879 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126225065 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚447437703981568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397368711634432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _step‚465 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6095172786688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5824564948754 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5845612363264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397368729637376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88625739 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 _runtime‚6066.718748962 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403215114645504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5846385008128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126248081 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387255760780624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5825335597166 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6095366928896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚393081113417150 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31614764 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚441342555134976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403214323997696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397368711634432 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99459816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88626295 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚393080325729378 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚387255777819984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99459814 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99459814 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5846385008128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225684879 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126225065 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚447437703981568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397368711634432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _step‚465 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6095172786688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5824564948754 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5845612363264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397368729637376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88625739 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 _runtime‚6066.718748962 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403215114645504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5846385008128 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126225065 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99459816 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126248081 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441342531194880ÑÌàñš2“- _wandbruntime‚6067  _runtime‚6067EtWù'”úM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225746508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441417903477760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403283819343872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 225723464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚397436429576704 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397436411573760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397436411573760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88640644 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚5699 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31620568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5826336737252 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57020625 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99476690 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6096215390720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚393148180992996 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5847389767168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚387321827216384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225746508 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126246776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57020623 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6096417908224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99476690 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5846617108992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126269818 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5846617108992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚447514094928384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚393147393305016 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884939680259643e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042¤Ã®'•ÿM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚447514094928384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚393147393305016 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884939680259643e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225746508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441417903477760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403283819343872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 225723464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚397436429576704 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397436411573760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397436411573760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88640644 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚5699 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31620568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5826336737252 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57020625 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99476690 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6096215390720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚393148180992996 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5847389767168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚387321827216384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225746508 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126246776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57020623 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6096417908224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99476690 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5846617108992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126269818 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5846617108992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600ÒPyZE2–- _wandbruntime‚6067  _runtime‚6067,}ÁÇ^—Ø /loss/FRt1/detection_loss‚2.6225496966664386 1loss/SRqqt3/detection_loss‚1.1784869430793656 epoch‚4 0loss/FRt1/assignment_loss‚3.4611348146345557 /loss/FRt3/assignment_loss‚2.000939903350977 2loss/SRqqt2/assignment_loss‚1.5974295638840308 0loss/FBt1/assignment_loss‚0.6089528103943436 1loss/FBt4/assignment_loss‚0.20854183679413738 2loss/SRqqt4/assignment_loss‚1.1068513133981521 /loss/FRt2/detection_loss‚2.4637316078454923 1loss/SRqqt4/detection_loss‚1.0643726746025293 &loss/total_loss‚41.864256219203455 % _timestamp‚1.7884939680292451e+09 2loss/SRqqt1/assignment_loss‚1.8043441443393629 0loss/FBt3/assignment_loss‚0.4748532408484607 /loss/FRt3/detection_loss‚1.8845440584879656 1loss/SRqqt1/detection_loss‚2.7119717597961426 /loss/FBt1/detection_loss‚2.4070426008918067 /loss/FBt4/detection_loss‚0.6019446443061571 .loss/FBt2/detection_loss‚2.068850127550272 /loss/FBt3/detection_loss‚2.5366570949554443 0loss/FRt4/assignment_loss‚1.6870206273519075 2loss/SRqqt3/assignment_loss‚1.1860958248635547 .loss/FRt4/detection_loss‚1.625259190511245 1loss/FBt2/assignment_loss‚0.46943397605704934 trainer/global_step‚5699 /loss/FRt2/assignment_loss‚3.792502916776217 1loss/SRqqt2/detection_loss‚2.3007448478178545 _runtime‚6067.669489552 _step‚467;™7¬c˜Ý % _timestamp‚1.7884939680292451e+09 2loss/SRqqt1/assignment_loss‚1.8043441443393629 0loss/FBt3/assignment_loss‚0.4748532408484607 /loss/FRt3/detection_loss‚1.8845440584879656 1loss/SRqqt1/detection_loss‚2.7119717597961426 /loss/FBt1/detection_loss‚2.4070426008918067 /loss/FBt4/detection_loss‚0.6019446443061571 .loss/FBt2/detection_loss‚2.068850127550272 /loss/FBt3/detection_loss‚2.5366570949554443 0loss/FRt4/assignment_loss‚1.6870206273519075 2loss/SRqqt3/assignment_loss‚1.1860958248635547 .loss/FRt4/detection_loss‚1.625259190511245 1loss/FBt2/assignment_loss‚0.46943397605704934 trainer/global_step‚5699 /loss/FRt2/assignment_loss‚3.792502916776217 1loss/SRqqt2/detection_loss‚2.3007448478178545 _runtime‚6067.669489552 _step‚467 /loss/FRt1/detection_loss‚2.6225496966664386 1loss/SRqqt3/detection_loss‚1.1784869430793656 epoch‚4 0loss/FRt1/assignment_loss‚3.4611348146345557 /loss/FRt3/assignment_loss‚2.000939903350977 2loss/SRqqt2/assignment_loss‚1.5974295638840308 0loss/FBt1/assignment_loss‚0.6089528103943436 1loss/FBt4/assignment_loss‚0.20854183679413738 2loss/SRqqt4/assignment_loss‚1.1068513133981521 /loss/FRt2/detection_loss‚2.4637316078454923 1loss/SRqqt4/detection_loss‚1.0643726746025293 &loss/total_loss‚41.864256219203455Ócƃß™‚(:’ —éÔùË’® gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.565' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.712999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cázš‚(:ï —éÔÊÞÛ°% disk./.usageGB‚1173.0550537109375% disk.nvme0n1.out‚150211.689453125" disk.dm-0.out‚150211.689453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2936.3046875 cpu‚25.244830453396975 network.sent‚ 1.3739311e+07) disk./.usagePercent‚78.72470891328939$ disk.nvme0c0n1.in‚22851.49609375 disk.drbd1260.out‚0.875 disk.sda.out‚ 180.0703125 disk.sda2.in‚0$ memory_percent‚93.17302703857422' disk.nvme0c0n1.out‚150211.689453125 disk.dm-0.in‚22851.49609375 disk.sda3.out‚ 180.0703125 disk.dm-1.in‚0 disk.dm-1.out‚ 180.0703125 disk.drbd1260.in‚0 disk.sda.in‚0& proc.memory.availableMB‚ 1118.53125 network.recv‚ 2.004975e+06" disk.nvme0n1.in‚22851.49609375 disk.sda3.in‚0* proc.memory.percent‚17.921781539916992 proc.cpu.threads‚17O¶Ù›‚(:’ ¦éÔ䘩¯ gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.148' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.629599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ufãzœ‚(:ï ¦éÔИ†±' disk.nvme0c0n1.out‚150216.419921875" disk.nvme0n1.in‚22851.49609375$ memory_percent‚93.17302703857422" proc.memory.rssMB‚ 2936.3046875 cpu‚25.244516111847854 disk.dm-1.out‚ 180.4609375 disk.drbd1260.out‚0.875 disk.sda2.out‚0 disk.dm-0.in‚22851.49609375* proc.memory.percent‚17.921781539916992 disk.sda3.in‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚22851.49609375% disk.nvme0n1.out‚150216.419921875" disk.dm-0.out‚150216.419921875& proc.memory.availableMB‚ 1118.53125 proc.cpu.threads‚17 disk.sda.out‚ 180.4609375 disk.sda1.out‚0 network.sent‚ 1.3834374e+07) disk./.usagePercent‚78.72578747303113% disk./.usageGB‚1173.0711250305176 disk.sda3.out‚ 180.4609375 disk.dm-1.in‚0 disk.sda.in‚0 network.recv‚ 2.010735e+06 disk.drbd1260.in‚0 disk.sda2.in‚0¼õqû’‚(:‡ µéÔÂŽÛ® gpu.0.gpu‚45.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊÚ‡ž‚(:ü µéÔ‰ñÔ²$ disk.nvme0c0n1.in‚22988.81640625) disk.nvme0c0n1.out‚150482.34521484375' disk.nvme0n1.out‚150482.34521484375 disk.sda3.in‚0 disk.dm-1.in‚0$ memory_percent‚93.17455291748047 network.sent‚ 1.3840679e+07 disk.sda2.in‚0 disk.dm-0.in‚22988.81640625$ disk.dm-0.out‚150482.34521484375 disk.dm-1.out‚ 181.08984375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.87890625 disk.sda.in‚0$ disk./.usageGB‚1173.133861541748" disk.nvme0n1.in‚22988.81640625 disk.sda3.out‚ 181.08984375 disk.sda1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 1118.28125" proc.memory.rssMB‚ 2936.3046875 proc.cpu.threads‚17 network.recv‚ 2.015498e+06 disk.sda.out‚ 181.08984375* proc.memory.percent‚17.921781539916992 cpu‚25.244111424413713) disk./.usagePercent‚78.72999777293926 disk.sda2.out‚0•2i2Ÿ- _wandbruntime‚6115  _runtime‚6115Èÿ=^Ç Á _runtime‚6115.107451598 _step‚468 & lr-AdamW/pg1‚0.0004979541096774899 & lr-AdamW/pg2‚0.0004979541096774899 trainer/global_step‚5749 % _timestamp‚1.7884940154670405e+09Ç Ì¡Æ _runtime‚6115.107451598 _step‚468 & lr-AdamW/pg1‚0.0004979541096774899 & lr-AdamW/pg2‚0.0004979541096774899 trainer/global_step‚5749 % _timestamp‚1.7884940154670405e+09ÔѤ”s2¢- _wandbruntime‚6115  _runtime‚6115µ)^|§ £ôO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5895864063488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100303514 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127336983 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406650363312128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚396434505790258 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 227617481 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚451257620027392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5874635533862 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100303516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400753726603776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400753708600832 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89375071 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31881694 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDevicÕgóSeStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400753708600832 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚445109972274176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6147857446912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 _step‚469 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406649572664320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 227640499 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185753600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚390559082568624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5896636708352 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚390559099607984 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57493375 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚6115.108709609 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400753726603776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚546 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 % _timestamp‚1.7884940154682522e+09Æ]‚2ÿ'¤ùO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 % _timestamp‚1.7884940154682522e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5895864063488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100303514 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127336983 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406650363312128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚396434505790258 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 227617481 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚451257620027392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5874635533862 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100303516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400753726603776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400753708600832 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89375071 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31881694 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400753708600832 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚445109972274176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6147857446912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 _step‚469 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406649572664320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 227640499 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185753600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚390559082568624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5896636708352 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚390559099607984 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57493375 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚6115.108709609 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400753726603776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚546Õ*eþi2¥- _wandbruntime‚6115  _runtime‚61155ºþs'¦ýM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100320390 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31886457 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5896868809216 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127335678 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚406718277349376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89389480 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5876407322360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚406719068010496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127335678 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396501573366104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100320388 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89390014 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227679110 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227656066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400821408540160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400821426543104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31886989 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 $ _timestamp‚1.788494016321236e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227679110 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390625166043744 _step‚470 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57503025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚396500785678124¬>ØV§‚N \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5876407322360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚406719068010496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127335678 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396501573366104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100320388 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89390014 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227679110 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227656066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400821408540160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400821426543104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31886989 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 $ _timestamp‚1.788494016321236e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227679110 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390625166043744 _step‚470 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57503025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚396500785678124 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100320390 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31886457 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5896868809216 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127335678 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚406718277349376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89389480 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352Ö¨‘²2¨- _wandbruntime‚6115  _runtime‚6115Ý¢LÞ^©Ø 1loss/SRqqt4/detection_loss‚1.0590520540832495 &loss/total_loss‚46.419133851460174 epoch‚4 0loss/FBt1/assignment_loss‚0.2663017660370315 1loss/SRqqt3/detection_loss‚1.7620878978017367 /loss/FBt1/detection_loss‚2.3780702352523804 /loss/FRt2/assignment_loss‚3.179352272640575 2loss/SRqqt1/assignment_loss‚1.6891494890784515 1loss/FBt4/assignment_loss‚0.07581645901189309 /loss/FRt3/detection_loss‚2.1322797853066477 .loss/FBt2/detection_loss‚9.375432670116425 0loss/FRt4/assignment_loss‚1.3463587793436917 2loss/SRqqt2/assignment_loss‚1.8537252121850063 /loss/FRt4/detection_loss‚1.3452085184780034 /loss/FBt4/detection_loss‚0.5348655423443568 _runtime‚6115.963917933 2loss/SRqqt3/assignment_loss‚0.9564540371439367 /loss/FBt3/detection_loss‚2.0103419944643974 _step‚471 1loss/SRqqt4/assignment_loss‚0.990732456397143 .loss/FRt1/detection_loss‚2.169119878941112 trainer/global_step‚5749 $ _timestamp‚1.788494016323646e+09 0loss/FRt3/assignment_loss‚1.8018049265063085 3loss/FBt2/assignment_loss‚0.0003296785580459982 1loss/FBt3/assignment_loss‚0.26614539004185644 /loss/FRt2/detection_loss‚2.5036570141760133 1loss/SRqqt1/detection_loss‚2.1672645658254623 0loss/SRqqt2/detection_loss‚3.115501981032522 /loss/FRt1/assignment_loss‚3.440081246693929ì¬C¸cªÝ /loss/FRt2/assignment_loss‚3.179352272640575 2loss/SRqqt1/assignment_loss‚1.6891494890784515 1loss/FBt4/assignment_loss‚0.07581645901189309 /loss/FRt3/detection_loss‚2.1322797853066477 .loss/FBt2/detection_loss‚9.375432670116425 0loss/FRt4/assignment_loss‚1.3463587793436917 2loss/SRqqt2/assignment_loss‚1.8537252121850063 /loss/FRt4/detection_loss‚1.3452085184780034 /loss/FBt4/detection_loss‚0.5348655423443568 _runtime‚6115.963917933 2loss/SRqqt3/assignment_loss‚0.9564540371439367 /loss/FBt3/detection_loss‚2.0103419944643974 _step‚471 1loss/SRqqt4/assignment_loss‚0.990732456397143 .loss/FRt1/detection_loss‚2.169119878941112 trainer/global_step‚5749 $ _timestamp‚1.788494016323646e+09 0loss/FRt3/assignment_loss‚1.8018049265063085 3loss/FBt2/assignment_loss‚0.0003296785580459982 1loss/FBt3/assignment_loss‚0.26614539004185644 /loss/FRt2/detection_loss‚2.5036570141760133 1loss/SRqqt1/detection_loss‚2.1672645658254623 0loss/SRqqt2/detection_loss‚3.115501981032522 /loss/FRt1/assignment_loss‚3.440081246693929 1loss/SRqqt4/detection_loss‚1.0590520540832495 &loss/total_loss‚46.419133851460174 epoch‚4 0loss/FBt1/assignment_loss‚0.2663017660370315 1loss/SRqqt3/detection_loss‚1.7620878978017367 /loss/FBt1/detection_loss‚2.3780702352523804×yË(«‚(:„ ÄéÔ•æ”® gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.86' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«LKȉ¬‚(:þ ÄéÔ¬‡’° disk.dm-1.in‚0 disk.dm-1.out‚ 181.56640625 disk.drbd1260.in‚0' proc.memory.availableMB‚ 1118.265625* proc.memory.percent‚17.921781539916992% disk./.usageGB‚1173.1961135864258$ disk.nvme0c0n1.in‚22988.81640625 disk.sda1.in‚0$ memory_percent‚93.17464828491211" proc.memory.rssMB‚ 2936.3046875 cpu‚25.243784249839056' disk.nvme0n1.out‚150760.13818359375 disk.sda.in‚0 disk.sda.out‚ 181.56640625 disk.sda1.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 1.3846307e+07) disk./.usagePercent‚78.73417555989053) disk.nvme0c0n1.out‚150760.13818359375 disk.dm-0.in‚22988.81640625$ disk.dm-0.out‚150760.13818359375 disk.sda3.out‚ 181.56640625 disk.drbd1260.out‚ 0.87890625 disk.sda2.in‚0 network.recv‚ 2.019332e+06" disk.nvme0n1.in‚22988.81640625 disk.sda3.in‚0#oË¿’­‚(:‡ ÓéÔŽ„Õ­ gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.056' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.8112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å•ü(†®‚(:û ÓéÔŸ—°$ disk./.usageGB‚1173.297695159912 disk.sda3.in‚0 disk.dm-1.out‚ 319.83984375 disk.drbd1260.out‚ 0.8828125 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 319.83984375) disk.nvme0c0n1.out‚151128.44775390625$ memory_percent‚93.17193031311035( proc.memory.availableMB‚ 1118.7109375" proc.memory.rssMB‚ 2936.3046875 disk.sda3.out‚ 319.83984375 disk.sda1.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚22988.81640625' disk.nvme0n1.out‚151128.44775390625 disk.dm-0.in‚22988.81640625* proc.memory.percent‚17.921781539916992 cpu‚25.24355839024584 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚22988.81640625$ disk.dm-0.out‚151128.44775390625 proc.cpu.threads‚17 network.sent‚ 1.3953633e+07 network.recv‚ 2.02507e+06) disk./.usagePercent‚78.74099278451982W/ßâ’¯‚(:‡ âéÔÑÿž­ gpu.0.gpu‚50.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.114' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆRË`‚°‚(:÷ âéÔìò®' proc.memory.availableMB‚ 1118.453125 proc.cpu.threads‚17 disk.dm-1.out‚ 326.015625 network.recv‚ 2.028571e+06" disk.nvme0n1.in‚22988.81640625 disk.dm-0.in‚22988.81640625 disk.dm-1.in‚0$ memory_percent‚93.17350387573242# proc.memory.rssMB‚ 2936.30859375 disk.sda3.in‚0 disk.drbd1260.out‚ 0.8828125 disk.sda1.in‚0 disk.sda2.out‚0 cpu‚25.243182532388204$ disk.nvme0c0n1.in‚22988.81640625' disk.nvme0n1.out‚151502.36962890625$ disk.dm-0.out‚151502.36962890625 disk.sda3.out‚ 326.015625 disk.drbd1260.in‚0 disk.sda.out‚ 326.015625 disk.sda1.out‚0* proc.memory.percent‚17.921805381774902 network.sent‚ 1.3958776e+07) disk./.usagePercent‚78.74110645186502$ disk./.usageGB‚1173.299388885498) disk.nvme0c0n1.out‚151502.36962890625 disk.sda.in‚0 disk.sda2.in‚0§'Tê2±- _wandbruntime‚6163  _runtime‚6163:…œÇ²Á & lr-AdamW/pg1‚0.0004924174219259413 & lr-AdamW/pg2‚0.0004924174219259413 trainer/global_step‚5799 % _timestamp‚1.7884940637514567e+09 _runtime‚6163.392034143 _step‚472ôÎ`Ì³Æ % _timestamp‚1.7884940637514567e+09 _runtime‚6163.392034143 _step‚472 & lr-AdamW/pg1‚0.0004924174219259413 & lr-AdamW/pg2‚0.0004924174219259413 trainer/global_step‚5799Ø;¾ÿ52´- _wandbruntime‚6163  _runtime‚6163Xûõú'µôO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101147214 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚404138705567232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128425885 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5946115763712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57975775 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32147369 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚448877389413376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229550083 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128402869 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5946115763712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚399787110475594 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101147214 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6200171315712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚393862404356624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5946888408576 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455077536788992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90123144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚393862421395984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚410084821330944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128402869 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005áv¶&¶ùO % _timestamp‚1.7884940637528121e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32147883 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 _step‚473 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213888000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 _runtime‚6163.393385169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128402869 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101147214 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚404138705567232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128425885 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5946115763712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57975775 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32147369 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚448877389413376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229550083 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128402869 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5946115763712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚399787110475594 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101147214 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6200171315712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚393862404356624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5946888408576 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455077536788992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90123144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚393862421395984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚410084821330944 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 _runtime‚6164.285424545 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229611712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚399854178051232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚404206405506560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5947120509440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5947120509440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57985425 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90138581 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101164090 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚448952713816064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101164088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚455153944634880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229588668 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚404206405506560 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32153156 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32152615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128424580 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚410154316677120 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90138038 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 % _timestamp‚1.7884940646439848e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚404206423509504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚410154316677120 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 trainer/global_step‚5799 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5947893167616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚399854965739212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229588668 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5947893167616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6201416559104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57985423 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚448952737756160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5926477907468 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393928470792384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5925707258848 _step‚474 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5926477907468 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393928470792384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5925707258848 _step‚474 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 _runtime‚6164.285424545 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229611712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚399854178051232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚404206405506560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5947120509440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5947120509440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57985425 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90138581 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101164090 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚448952713816064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101164088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚455153944634880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229588668 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚404206405506560 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32153156 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32152615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128424580 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚410154316677120 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90138038 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 % _timestamp‚1.7884940646439848e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚404206423509504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚410154316677120 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 trainer/global_step‚5799 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5947893167616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚399854965739212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229588668 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5947893167616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6201416559104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57985423 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005Úý ~2º- _wandbruntime‚6164  _runtime‚6164%%¤žS»Í /loss/SRqqt2/assignment_loss‚1.3237732302758 2loss/SRqqt4/assignment_loss‚1.0830422374307207 .loss/FBt3/detection_loss‚2.888643652200699 epoch‚4 % _timestamp‚1.7884940646472054e+09 _runtime‚6164.287394157 1loss/SRqqt4/detection_loss‚0.9151171437434612 1loss/FBt3/assignment_loss‚0.49154392836267863 /loss/FRt4/detection_loss‚2.0403158095338054 0loss/FRt4/assignment_loss‚1.6646497861759082 /loss/FRt1/detection_loss‚2.1275686383247376 .loss/FRt2/detection_loss‚2.390117808133729 1loss/SRqqt3/detection_loss‚1.3553857319056988 %loss/total_loss‚40.81291499337606 _step‚475 /loss/FRt1/assignment_loss‚3.623225671052933 /loss/FRt3/assignment_loss‚1.707511307163672 1loss/SRqqt3/assignment_loss‚1.023265016824007 .loss/FBt1/detection_loss‚2.107413785798209 /loss/FBt4/detection_loss‚0.5849991220179512 0loss/FBt2/assignment_loss‚0.2060745364333343 .loss/FRt3/detection_loss‚2.028862840750001 0loss/SRqqt2/detection_loss‚2.250381629313192 1loss/FBt4/assignment_loss‚0.18851653740199084 2loss/SRqqt1/assignment_loss‚1.4090160822326487 /loss/FBt1/assignment_loss‚0.623629389929176 trainer/global_step‚5799 /loss/FRt2/assignment_loss‚3.446766287088394 0loss/SRqqt1/detection_loss‚2.373169214436502 /loss/FBt2/detection_loss‚2.9599256068468094ÃÊôuX¼Ò 0loss/FRt4/assignment_loss‚1.6646497861759082 /loss/FRt1/detection_loss‚2.1275686383247376 .loss/FRt2/detection_loss‚2.390117808133729 1loss/SRqqt3/detection_loss‚1.3553857319056988 %loss/total_loss‚40.81291499337606 _step‚475 /loss/FRt1/assignment_loss‚3.623225671052933 /loss/FRt3/assignment_loss‚1.707511307163672 1loss/SRqqt3/assignment_loss‚1.023265016824007 .loss/FBt1/detection_loss‚2.107413785798209 /loss/FBt4/detection_loss‚0.5849991220179512 0loss/FBt2/assignment_loss‚0.2060745364333343 .loss/FRt3/detection_loss‚2.028862840750001 0loss/SRqqt2/detection_loss‚2.250381629313192 1loss/FBt4/assignment_loss‚0.18851653740199084 2loss/SRqqt1/assignment_loss‚1.4090160822326487 /loss/FBt1/assignment_loss‚0.623629389929176 trainer/global_step‚5799 /loss/FRt2/assignment_loss‚3.446766287088394 0loss/SRqqt1/detection_loss‚2.373169214436502 /loss/FBt2/detection_loss‚2.9599256068468094 /loss/SRqqt2/assignment_loss‚1.3237732302758 2loss/SRqqt4/assignment_loss‚1.0830422374307207 .loss/FBt3/detection_loss‚2.888643652200699 epoch‚4 % _timestamp‚1.7884940646472054e+09 _runtime‚6164.287394157 1loss/SRqqt4/detection_loss‚0.9151171437434612 1loss/FBt3/assignment_loss‚0.49154392836267863 /loss/FRt4/detection_loss‚2.0403158095338054Û€”Àÿw½‚(:ì ñéÔ°©Õ°! disk.dm-0.out‚151729.14453125) proc.memory.availableMB‚ 1118.66796875 network.sent‚ 1.3978061e+07 disk.drbd1260.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 23153.3984375$ memory_percent‚93.17219257354736# proc.memory.rssMB‚ 2936.30859375 cpu‚25.242783787333043 proc.cpu.threads‚17 network.recv‚ 2.033394e+06% disk./.usageGB‚1172.5144805908203 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.88671875 disk.sda2.out‚0$ disk.nvme0n1.out‚151729.14453125 disk.dm-0.in‚ 23153.3984375) disk./.usagePercent‚78.68843059762727 disk.sda3.out‚ 498.90625 disk.sda.in‚0! disk.nvme0n1.in‚ 23153.3984375* proc.memory.percent‚17.921805381774902 disk.sda3.in‚0 disk.dm-1.out‚ 498.90625 disk.sda.out‚ 498.90625 disk.sda1.in‚0 disk.sda2.in‚0& disk.nvme0c0n1.out‚151729.14453125Ä„€’¾‚(:‡ ñéÔ—¯ñ° gpu.0.gpu‚36.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.623' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mp; ’¿‚(:‡ €‚éÔׇƭ gpu.0.gpu‚38.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.239' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0478 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Š1À€À‚(:õ €‚éÔŠŽ°¯ network.sent‚ 1.4073977e+07) disk./.usagePercent‚78.67901950466106 disk.dm-1.out‚ 499.37890625 disk.drbd1260.out‚ 0.88671875 disk.sda2.in‚0" disk.nvme0n1.in‚23153.83984375* proc.memory.percent‚17.921805381774902 cpu‚25.242372641654207 network.recv‚ 2.040242e+06 disk.sda3.out‚ 499.37890625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚23153.83984375% disk.nvme0c0n1.out‚152194.7734375% disk./.usageGB‚1172.3742485046387 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0# disk.nvme0n1.out‚152194.7734375 disk.dm-0.in‚23153.83984375 disk.dm-0.out‚152194.7734375# proc.memory.rssMB‚ 2936.30859375 disk.sda.out‚ 499.37890625$ memory_percent‚93.17371845245361) proc.memory.availableMB‚ 1118.41796875 proc.cpu.threads‚17¦F0Á‚(:’ ‚éԭ㥮 gpu.0.gpu‚45.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.635' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.127000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èj.‚‚(:÷ ‚éÔ¡ÇØ® disk.sda2.in‚0$ disk.nvme0c0n1.in‚23153.83984375 disk.dm-0.in‚23153.83984375$ disk.dm-0.out‚152303.77197265625 proc.cpu.threads‚17 disk.sda1.in‚0% disk./.usageGB‚1172.8728790283203 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0" disk.nvme0n1.in‚23153.83984375# memory_percent‚93.1736707687378) proc.memory.availableMB‚ 1118.42578125) disk./.usagePercent‚78.71248301748422 disk.sda3.out‚ 499.765625 disk.sda1.out‚0 disk.sda2.out‚0' disk.nvme0n1.out‚152303.77197265625# proc.memory.rssMB‚ 2936.30859375 disk.sda.out‚ 499.765625 network.sent‚ 1.4079214e+07 network.recv‚ 2.04435e+06 disk.dm-1.out‚ 499.765625 disk.drbd1260.out‚0.890625) disk.nvme0c0n1.out‚152303.77197265625* proc.memory.percent‚17.921805381774902 cpu‚25.241958325980345 disk.sda3.in‚0˜•±l2Ã- _wandbruntime‚6208  _runtime‚6208’?bIÇÄÁ trainer/global_step‚5849 % _timestamp‚1.7884941085238714e+09 _runtime‚6208.164244498 _step‚476 & lr-AdamW/pg1‚0.0004868606661556758 & lr-AdamW/pg2‚0.0004868606661556758ÅÆY²ÌÅÆ & lr-AdamW/pg2‚0.0004868606661556758 trainer/global_step‚5849 % _timestamp‚1.7884941085238714e+09 _runtime‚6208.164244498 _step‚476 & lr-AdamW/pg1‚0.0004868606661556758Ü ³2Æ- _wandbruntime‚6208  _runtime‚6208,£ÝKìÇôO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚413520860645376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5996367463936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6252929502720 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚407523702533632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90873743 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32415012 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5997140108800 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101990914 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32415566 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6252737457664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231482685 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101990914 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5997140108800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 _runtime‚6208.165750475 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5996367463936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚413520069997568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚458897520070144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚397165743183984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚413520860645376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129514787 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚407523720536576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_splÛ}Øvit_bytes.small_pool.current‚ 192045056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58458177 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚397165726144624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231505703 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5974776704078 % _timestamp‚1.7884941085252976e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚413520069997568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101990916 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129491771 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101990916 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚477 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚452644782612480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚458897736055296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚403140502848702 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90873187 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328ë³"¶ÿ'ÈùO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90873187 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚413520860645376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5996367463936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6252929502720 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚407523702533632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90873743 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32415012 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5997140108800 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101990914 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32415566 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6252737457664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231482685 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101990914 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5997140108800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 _runtime‚6208.165750475 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5996367463936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚413520069997568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚458897520070144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚397165743183984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚413520860645376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129514787 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚407523720536576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58458177 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚397165726144624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231505703 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5974776704078 % _timestamp‚1.7884941085252976e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚413520069997568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101990916 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129491771 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101990916 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚477 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚452644782612480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚458897736055296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚403140502848702Ý1«÷2É- _wandbruntime‚6209  _runtime‚6209î\'ÊþM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102007788 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚407591420475904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102007790 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5998144867840 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129536524 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129513482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚458973925503488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚403207570424340 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5997372209664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102007790 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129513482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚407591402472960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58467825 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚397231809619744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚397231792580384 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231544314 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5998144867840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90888163 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚403208358112320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚407591402472960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5975777843956 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231521270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129536524 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚413588774682624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58467823 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6253794548224 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5997372209664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884941094022188e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102007788 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90888728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452720130955264 _step‚478 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚407591420475904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5976548492576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044C°´ÓšËƒN ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58467823 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6253794548224 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5997372209664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884941094022188e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102007788 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90888728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452720130955264 _step‚478 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚407591420475904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5976548492576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102007788 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚407591420475904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102007790 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5998144867840 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129536524 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129513482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚458973925503488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚403207570424340 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5997372209664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102007790 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129513482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚407591402472960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58467825 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚397231809619744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚397231792580384 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231544314 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 599814486784A#\Èo 0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90888163 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚403208358112320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚407591402472960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5975777843956 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231521270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129536524 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚413588774682624Þ­2\Å2Ì- _wandbruntime‚6209  _runtime‚6209Ù½XÍÒ .loss/FRt1/detection_loss‚2.783593351350111 1loss/SRqqt4/detection_loss‚0.9989219732203726 /loss/FBt1/detection_loss‚2.3578937544542202 %loss/total_loss‚42.52310913677061 /loss/FRt2/assignment_loss‚3.531115061527974 2loss/FBt4/assignment_loss‚0.044340040630753516 2loss/SRqqt4/assignment_loss‚0.9382835726858086 0loss/SRqqt1/detection_loss‚2.317691842360156 epoch‚4 2loss/SRqqt1/assignment_loss‚1.4915424258049046 0loss/FBt1/assignment_loss‚0.8447276251685895 1loss/FBt3/assignment_loss‚0.43560277167125605 trainer/global_step‚5849 _step‚479 1loss/SRqqt2/assignment_loss‚1.751702487894467 .loss/FBt2/detection_loss‚2.649139037499061 -loss/FBt3/detection_loss‚2.24249579757452 0loss/FRt1/assignment_loss‚3.5130703659618603 1loss/SRqqt3/assignment_loss‚1.311535898535573 0loss/FBt2/assignment_loss‚0.3280816375604795 1loss/SRqqt2/detection_loss‚2.5327953917639596 1loss/SRqqt3/detection_loss‚1.3898223408243873 $ _timestamp‚1.788494109404513e+09 0loss/FRt3/assignment_loss‚2.5851043853908777 /loss/FRt2/detection_loss‚2.4177397399235376 .loss/FBt4/detection_loss‚0.681002674321884 _runtime‚6209.044709014 0loss/FRt4/assignment_loss‚1.4175867619721785 /loss/FRt3/detection_loss‚1.9927265474107116 /loss/FRt4/detection_loss‚1.9665936512629623RÐÝ]Î× 2loss/SRqqt4/assignment_loss‚0.9382835726858086 0loss/SRqqt1/detection_loss‚2.317691842360156 epoch‚4 2loss/SRqqt1/assignment_loss‚1.4915424258049046 0loss/FBt1/assignment_loss‚0.8447276251685895 1loss/FBt3/assignment_loss‚0.43560277167125605 trainer/global_step‚5849 _step‚479 1loss/SRqqt2/assignment_loss‚1.751702487894467 .loss/FBt2/detection_loss‚2.649139037499061 -loss/FBt3/detection_loss‚2.24249579757452 0loss/FRt1/assignment_loss‚3.5130703659618603 1loss/SRqqt3/assignment_loss‚1.311535898535573 0loss/FBt2/assignment_loss‚0.3280816375604795 1loss/SRqqt2/detection_loss‚2.5327953917639596 1loss/SRqqt3/detection_loss‚1.3898223408243873 $ _timestamp‚1.788494109404513e+09 0loss/FRt3/assignment_loss‚2.5851043853908777 /loss/FRt2/detection_loss‚2.4177397399235376 .loss/FBt4/detection_loss‚0.681002674321884 _runtime‚6209.044709014 0loss/FRt4/assignment_loss‚1.4175867619721785 /loss/FRt3/detection_loss‚1.9927265474107116 /loss/FRt4/detection_loss‚1.9665936512629623 .loss/FRt1/detection_loss‚2.783593351350111 1loss/SRqqt4/detection_loss‚0.9989219732203726 /loss/FBt1/detection_loss‚2.3578937544542202 %loss/total_loss‚42.52310913677061 /loss/FRt2/assignment_loss‚3.531115061527974 2loss/FBt4/assignment_loss‚0.044340040630753516ßÔ‡“l’Ï‚(:‡ ž‚éÔɪ±¯ gpu.0.gpu‚38.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.066' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áºÎDЂ(:ö ž‚éÔ×Ù˱ disk.dm-1.out‚ 500.359375 disk.drbd1260.out‚0.890625" disk.nvme0n1.in‚23153.83984375' disk.nvme0n1.out‚152613.13134765625$ disk.dm-0.out‚152613.13134765625 cpu‚25.241556233567444 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 500.359375 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2936.30859375* proc.memory.percent‚17.921805381774902 network.sent‚ 1.408423e+07 disk.sda3.out‚ 500.359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚23153.83984375' proc.memory.availableMB‚ 1117.890625 network.recv‚ 2.047955e+06% disk./.usageGB‚1173.0045471191406 disk.sda.in‚0$ disk.nvme0c0n1.in‚23153.83984375) disk.nvme0c0n1.out‚152613.13134765625$ memory_percent‚93.17693710327148 proc.cpu.threads‚17) disk./.usagePercent‚78.72131937353605 disk.sda3.in‚0œB/k‘Ñ‚(:† ­‚éÔ¸œ£­ gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.525' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.105 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û~N*‹Ò‚(:€ ­‚éÔêö£° disk.dm-0.in‚23153.83984375 disk.drbd1260.out‚ 0.89453125' disk.nvme0n1.out‚153010.58056640625* proc.memory.percent‚17.921805381774902 cpu‚25.241190437810324 proc.cpu.threads‚17 disk.sda3.out‚ 500.85546875 disk.dm-1.out‚ 500.85546875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚153010.58056640625) proc.memory.availableMB‚ 1118.11328125 network.sent‚ 1.4193482e+07 network.recv‚ 2.055063e+06$ disk.dm-0.out‚153010.58056640625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 500.85546875) disk./.usagePercent‚78.73177472123442 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚23153.83984375" disk.nvme0n1.in‚23153.83984375# memory_percent‚93.1755781173706# proc.memory.rssMB‚ 2936.30859375% disk./.usageGB‚1173.1603393554688“zdÚˆÓ‚(:ý ¼‚éÔ‡ý«± network.sent‚ 1.4199384e+07 network.recv‚ 2.059442e+06) disk./.usagePercent‚78.73332049472617" disk.nvme0n1.in‚23329.97265625 disk.dm-0.in‚23329.97265625 disk.sda.out‚ 501.4140625$ memory_percent‚93.17409992218018 proc.cpu.threads‚17 disk.sda1.in‚0) disk.nvme0c0n1.out‚153392.04931640625' disk.nvme0n1.out‚153392.04931640625$ disk.dm-0.out‚153392.04931640625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 501.4140625% disk./.usageGB‚1173.1833724975586 disk.sda1.out‚0$ disk.nvme0c0n1.in‚23329.97265625 disk.sda3.out‚ 501.4140625 disk.sda.in‚0) proc.memory.availableMB‚ 1118.35546875# proc.memory.rssMB‚ 2936.30859375* proc.memory.percent‚17.921805381774902 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.89453125 cpu‚25.24080772280573úŽU’Ô‚(:‡ ¼‚éÔæñ™¸ gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.352' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.8704 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šŒœÕ‚(:‘ Ë‚éÔà™¡¯ gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚128.29' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.657999999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù÷® ~Ö‚(:ó Ë‚éÔ¤”…¶# proc.memory.rssMB‚ 2936.30859375 cpu‚25.240553251173047 disk.sda3.out‚ 501.7265625 disk.sda.out‚ 501.7265625' proc.memory.availableMB‚ 1117.828125* proc.memory.percent‚17.921805381774902 proc.cpu.threads‚17 network.sent‚ 1.4204542e+07 network.recv‚ 2.063014e+06 disk.sda3.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚153656.888671875" disk.nvme0n1.in‚23329.97265625 disk.dm-0.in‚23329.97265625$ memory_percent‚93.17731857299805( disk./.usagePercent‚78.7316303329851$ disk./.usageGB‚1173.158187866211 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚23329.97265625% disk.nvme0n1.out‚153656.888671875 disk.dm-1.out‚ 501.7265625 disk.drbd1260.out‚ 0.8984375 disk.sda.in‚0 disk.sda2.in‚0" disk.dm-0.out‚153656.888671875ªš—2×- _wandbruntime‚6260  _runtime‚6260m­G›ÅØ¿ _runtime‚6260.084615925 _step‚480 % lr-AdamW/pg1‚0.000481285000386267 % lr-AdamW/pg2‚0.000481285000386267 trainer/global_step‚5899 % _timestamp‚1.7884941604440386e+09·ø^ÊÙÄ _step‚480 % lr-AdamW/pg1‚0.000481285000386267 % lr-AdamW/pg2‚0.000481285000386267 trainer/global_step‚5899 % _timestamp‚1.7884941604440386e+09 _runtime‚6260.084615925à(÷Öé2Ú- _wandbruntime‚6260  _runtime‚62602з@ù'ÛóO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚416956109312000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462717404535808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _step‚481 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6047391809024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462717626812416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130580673 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚410908717502976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400469047932624 % _timestamp‚1.7884941604458854e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6046619164160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91622899 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 _runtime‚ 6260.08645565 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58940575 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102834616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233415287 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102834614 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32682322 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102834616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456412199751680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6024847289186 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400469064971984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚416955318664192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58940577 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚410908717502976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚416955318664192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102834614 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚456412223691776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6305403120640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚410908699500032 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130603689 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130580673 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 trainer/global_step‚5899 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91622349 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32681774 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚410908699500032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6047391809024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6305204784128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚406494682909582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0  ½èþ'ÜøO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462717404535808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _step‚481 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6047391809024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462717626812416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130580673 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚410908717502976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400469047932624 % _timestamp‚1.7884941604458854e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6046619164160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91622899 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 _runtime‚ 6260.08645565 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58940575 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102834616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233415287 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102834614 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32682322 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102834616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456412199751680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6024847289186 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400469064971984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚416955318664192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58940577 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚410908717502976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚416955318664192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102834614 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚456412223691776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6305403120640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚410908699500032 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130603689 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130580673 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 trainer/global_step‚5899 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91622349 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32681774 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚410908699500032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6047391809024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6305204784128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚406494682909582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚416956109312000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412á=Üá2Ý- _wandbruntime‚6261  _runtime‚6261@„òÚŸÞþM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 % _timestamp‚1.7884941614327126e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6047623909888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _step‚482 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚410976399439360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6025848429064 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417024814010368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102851488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6306452670464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58950223 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6047623909888 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130625426 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130602384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚400535114368384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130602384 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233476916 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91637286 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233476916 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚410976399439360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚410976417442304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187837440 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚406561750485428 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚410976417442304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102851488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚456487572034560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5899 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102851490 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417024023349248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462794024705024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6306264833024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0v࣠'߃N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚410976399439360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚410976417442304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187837440 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚406561750485428 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚410976417442304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102851488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚456487572034560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5899 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102851490 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417024023349248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462794024705024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6306264833024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 % _timestamp‚1.7884941614327126e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6047623909888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _step‚482 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚410976399439360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6025848429064 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417024814010368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102851488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6306452670464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58950223 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6047623909888 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130625426 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130602384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚400535114368384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130602384 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233476916 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91637286 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233476916 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0âG«!í2à- _wandbruntime‚6261  _runtime‚62614à¸\áÖ 2loss/SRqqt1/assignment_loss‚1.3381985599218413 2loss/SRqqt3/assignment_loss‚0.8964699396477421 /loss/FBt3/detection_loss‚2.6190761804580687 %loss/total_loss‚39.34768236983369 0loss/FRt2/assignment_loss‚2.6747519850730894 2loss/SRqqt4/assignment_loss‚1.1505548917394999 /loss/FRt1/detection_loss‚2.2621800225973128 1loss/SRqqt3/detection_loss‚1.3546131431783845 _runtime‚ 6261.07593656 0loss/FRt1/assignment_loss‚3.2794177746772766 0loss/FRt3/assignment_loss‚2.0633777684353767 /loss/FRt2/detection_loss‚2.5736472930227023 .loss/FRt4/detection_loss‚1.819477573228188 1loss/SRqqt2/detection_loss‚2.3422967586666346 _step‚483 1loss/FBt4/assignment_loss‚0.19876219962593192 1loss/SRqqt4/detection_loss‚0.8836903357750749 /loss/FBt2/detection_loss‚2.4608640330178395 epoch‚4 trainer/global_step‚5899 0loss/FRt4/assignment_loss‚0.8624100975501232 /loss/FBt1/detection_loss‚3.4548197746276856 $ _timestamp‚1.788494161435715e+09 1loss/SRqqt2/assignment_loss‚1.604417298687622 /loss/FRt3/detection_loss‚1.9234437085291807 1loss/SRqqt1/detection_loss‚2.0505589156649835 0loss/FBt1/assignment_loss‚0.3668565850239247 1loss/FBt2/assignment_loss‚0.10651921846770815 0loss/FBt3/assignment_loss‚0.3598333090242704 .loss/FBt4/detection_loss‚0.701445003193229“¤ uaâÛ 1loss/SRqqt4/detection_loss‚0.8836903357750749 /loss/FBt2/detection_loss‚2.4608640330178395 epoch‚4 trainer/global_step‚5899 0loss/FRt4/assignment_loss‚0.8624100975501232 /loss/FBt1/detection_loss‚3.4548197746276856 $ _timestamp‚1.788494161435715e+09 1loss/SRqqt2/assignment_loss‚1.604417298687622 /loss/FRt3/detection_loss‚1.9234437085291807 1loss/SRqqt1/detection_loss‚2.0505589156649835 0loss/FBt1/assignment_loss‚0.3668565850239247 1loss/FBt2/assignment_loss‚0.10651921846770815 0loss/FBt3/assignment_loss‚0.3598333090242704 .loss/FBt4/detection_loss‚0.701445003193229 2loss/SRqqt1/assignment_loss‚1.3381985599218413 2loss/SRqqt3/assignment_loss‚0.8964699396477421 /loss/FBt3/detection_loss‚2.6190761804580687 %loss/total_loss‚39.34768236983369 0loss/FRt2/assignment_loss‚2.6747519850730894 2loss/SRqqt4/assignment_loss‚1.1505548917394999 /loss/FRt1/detection_loss‚2.2621800225973128 1loss/SRqqt3/detection_loss‚1.3546131431783845 _runtime‚ 6261.07593656 0loss/FRt1/assignment_loss‚3.2794177746772766 0loss/FRt3/assignment_loss‚2.0633777684353767 /loss/FRt2/detection_loss‚2.5736472930227023 .loss/FRt4/detection_loss‚1.819477573228188 1loss/SRqqt2/detection_loss‚2.3422967586666346 _step‚483 1loss/FBt4/assignment_loss‚0.19876219962593192ãÖuãþ’ã‚(:‡ Ú‚éÔ©‰õ² gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.354' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü‚‘ã}ä‚(:ò Ú‚éÔ–™Ž¶ proc.cpu.threads‚17 network.recv‚ 2.066419e+06$ disk.nvme0c0n1.in‚23329.97265625 disk.sda3.in‚0 disk.sda3.out‚ 502.203125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.921805381774902 network.sent‚ 1.4209587e+07$ disk./.usageGB‚1173.183578491211% disk.nvme0n1.out‚153843.451171875 disk.dm-0.in‚23329.97265625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 502.203125 disk.sda2.out‚0) disk./.usagePercent‚78.73333431913301' disk.nvme0c0n1.out‚153843.451171875 disk.drbd1260.out‚ 0.8984375 disk.sda2.in‚0$ memory_percent‚93.17753314971924) proc.memory.availableMB‚ 1117.79296875# proc.memory.rssMB‚ 2936.30859375" disk.nvme0n1.in‚23329.97265625" disk.dm-0.out‚153843.451171875 disk.dm-1.out‚ 502.203125 cpu‚25.24037866533531À®_å‚(:’ é‚éÔ”½­ gpu.0.gpu‚42.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.743' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.548599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æTd¤‚æ‚(:÷ é‚éÔ°äÕ¯$ disk.nvme0c0n1.in‚23329.97265625* proc.memory.percent‚17.921805381774902 network.sent‚ 1.431881e+07 disk.dm-1.out‚ 502.578125 disk.drbd1260.out‚ 0.90234375" disk.nvme0n1.in‚23329.97265625 proc.cpu.threads‚17 disk.dm-0.in‚23329.97265625$ disk.dm-0.out‚153961.92041015625 disk.sda3.out‚ 502.578125 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚153961.92041015625$ memory_percent‚93.17598342895508 network.recv‚ 2.073198e+06 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0' disk.nvme0n1.out‚153961.92041015625' proc.memory.availableMB‚ 1118.046875# proc.memory.rssMB‚ 2936.30859375 cpu‚25.24002195094236) disk./.usagePercent‚78.72261195557643% disk./.usageGB‚1173.0238075256348 disk.sda3.in‚0 disk.sda.out‚ 502.578125 disk.sda2.out‚0°*#’ç‚(:‡ ø‚éÔ؆ø¯ gpu.0.gpu‚36.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.2062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™&´yè‚(:î ø‚éÔê´± cpu‚25.239641484296378 network.recv‚ 2.077842e+06( disk./.usagePercent‚78.7364834678117 disk.nvme0c0n1.in‚ 23484.6875 disk.dm-0.in‚ 23484.6875 disk.dm-1.in‚0 disk.dm-1.out‚ 503.10546875 disk.drbd1260.out‚ 0.90234375 disk.sda.out‚ 503.10546875% disk./.usageGB‚1173.2305030822754 disk.nvme0n1.in‚ 23484.6875% disk.nvme0n1.out‚154143.337890625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0$ memory_percent‚93.17600727081299) proc.memory.availableMB‚ 1118.04296875 network.sent‚ 1.4325016e+07" disk.dm-0.out‚154143.337890625 disk.sda1.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2936.30859375 proc.cpu.threads‚17' disk.nvme0c0n1.out‚154143.337890625 disk.sda3.out‚ 503.10546875 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.921805381774902}S2é- _wandbruntime‚6307  _runtime‚6307Zj¥Éêà % _timestamp‚1.7884942082340782e+09 _runtime‚6307.874485878 _step‚484 ' lr-AdamW/pg1‚0.00047569158657810443 ' lr-AdamW/pg2‚0.00047569158657810443 trainer/global_step‚5949Üì³¥ÎëÈ trainer/global_step‚5949 % _timestamp‚1.7884942082340782e+09 _runtime‚6307.874485878 _step‚484 ' lr-AdamW/pg1‚0.00047569158657810443 ' lr-AdamW/pg2‚0.00047569158657810443ääøÇ2ì- _wandbruntime‚6307  _runtime‚6307’¾Ñ’ÿíôO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131669575 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6307.875764506 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚466537595987456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103678314 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚420390567330816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚485 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420390567330816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚414293714469376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚414293696466432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59422977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6096870864384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6357758917120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131669575 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32949311 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 trainer/global_step‚5949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420391357978624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚460179640830976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59422975 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403772386759984 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6074917874294 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103678314 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92372286 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6096870864384 MFDeviceâ<ÂûStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409847287594918 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6097643509248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414293714469376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6075688522706 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚420391357978624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6097643509248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92372836 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235370907 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103678316 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6357955156480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409848075282690 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460179616890880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131692591ÄÛ¼3ÿ'îùO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32949859 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131692591 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 % _timestamp‚1.7884942082352378e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103678316 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414293696466432 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420390567330816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚414293714469376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚414293696466432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59422977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6096870864384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6357758917120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131669575 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32949311 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 trainer/global_step‚5949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420391357978624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚460179640830976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59422975 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403772386759984 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6074917874294 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103678314 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92372286 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6096870864384 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409847287594918 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6097643509248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414293714469376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6075688522706 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚420391357978624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6097643509248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92372836 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235370907 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103678316 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6357955156480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409848075282690 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460179616890880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131692591 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131669575 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6307.875764506 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚466537595987456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103678314 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚420390567330816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚485 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚409914355170556 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚403838453195744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235409518 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6098648268288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59432623 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚466613766306304 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92387236 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59432625 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 $ _timestamp‚1.788494209168651e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚414361396405760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32955171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚460254989173760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420460062676992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6076689662792 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚558 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235386474 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235386474 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6075919014172 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466613990666752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6359001492992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6097875610112 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103695188 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32954613 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6358801072640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 _step‚486 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚460254965233664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409915142858536 _runtime‚6308.809775885 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235409518 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6097875610112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403838436156384 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103695188 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚409914355170556 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚403838453195744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235409518 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6098648268288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59432623 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚466613766306304 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92387236 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59432625 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 $ _timestamp‚1.788494209168651e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚414361396405760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32955171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚460254989173760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420460062676992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6076689662792 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚558 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235386474 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235386474 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6075919014172 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466613990666752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6359001492992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6097875610112 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103695188 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32954613 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6358801072640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 _step‚486 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚460254965233664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409915142858536 _runtime‚6308.809775885 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235409518 FDeviceStatsMonitor.on_train_batch_end/allo༅Zcated_bytes.all.peak‚ 11892004352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6097875610112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403838436156384 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103695188 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230665216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407075691508624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33216956 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423826606645248 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104522014 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6410685184000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423825815997440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6410478458880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93122331 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423825815997440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417678711435776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6147122564608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚413201467655798 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚463947034030080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417678693432832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _runtime‚6359.547124154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 trainer/global_step‚5999 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423826606645248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132758477 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6125759107814 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93122886 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33217509 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6124988459402 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚413200679968026 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104522014 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206725120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6410685184000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423825815997440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6410478458880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93122331 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423825815997440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417678711435776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6147122564608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚413201467655798 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚463947034030080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417678693432832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _runtime‚6359.547124154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 trainer/global_step‚5999 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423826606645248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132758477 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6125759107814 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93122886 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33217509 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6124988459402 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚413200679968026 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104522014 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206725120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230665216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407075691508624éndLÌ’‚(:‡ ´ƒéÔӧ̮ gpu.0.gpu‚43.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øê ƒ‚‚(:ø ´ƒéÔ¢ò×° disk.sda.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 23621.078125& disk.nvme0n1.out‚155532.9853515625$ memory_percent‚93.17786693572998 cpu‚25.238304937659965 network.sent‚ 1.4450323e+07% disk./.usageGB‚1172.7108421325684 disk.dm-1.out‚ 504.81640625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.91015625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 2.096183e+06 disk.sda3.in‚0 disk.sda3.out‚ 504.81640625 disk.sda.out‚ 504.81640625( disk.nvme0c0n1.out‚155532.9853515625 disk.nvme0n1.in‚ 23621.078125 disk.dm-0.in‚ 23621.078125# disk.dm-0.out‚155532.9853515625) disk./.usagePercent‚78.70160858545229) proc.memory.availableMB‚ 1117.73828125# proc.memory.rssMB‚ 2936.30859375* proc.memory.percent‚17.921805381774902 proc.cpu.threads‚17 disk.dm-1.in‚0ŒšEö2ƒ- _wandbruntime‚6360  _runtime‚6360åuNÊ„þM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423895311343616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93137216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407141774983744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6148127310336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417746393372160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93137776 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6148127310336 trainer/global_step‚5999 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104538888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current<#€¬:‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104538890 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417746411375104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417746393372160 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚470433912659968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237342120 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104538890 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464022382372864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132803230 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423895311343616 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚558 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59915025 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6411530287104 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 % _timestamp‚1.7884942607736154e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132780188 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 237342120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6411726513152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚413268535231644 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413267747543664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407141757944384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6148899968512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 237319076 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846¬®W '…ƒN HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104538890 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417746411375104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417746393372160 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚470433912659968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237342120 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104538890 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464022382372864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132803230 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423895311343616 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚558 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59915025 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6411530287104 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 % _timestamp‚1.7884942607736154e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132780188 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 237342120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6411726513152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚413268535231644 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413267747543664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407141757944384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6148899968512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 237319076 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423895311343616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93137216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407141774983744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6148127310336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417746393372160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93137776 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6148127310336 trainer/global_step‚5999 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104538888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489êî)2†- _wandbruntime‚6360  _runtime‚6360úkofa‡Û /loss/FRt3/detection_loss‚2.2568432252581525 1loss/SRqqt2/detection_loss‚2.3600781206041574 _step‚491 0loss/FRt4/assignment_loss‚1.2682880597809951 2loss/SRqqt2/assignment_loss‚1.7680761869996786 1loss/FBt1/assignment_loss‚0.48183159969214884 /loss/FRt1/detection_loss‚2.3617098952861544 /loss/FBt1/detection_loss‚2.8669481916087016 % _timestamp‚1.7884942607758744e+09 /loss/FBt3/detection_loss‚2.6099713921546934 2loss/SRqqt4/assignment_loss‚1.2121739398639697 1loss/SRqqt3/detection_loss‚1.4870544701578126 /loss/FBt2/detection_loss‚2.0560434886387418 0loss/SRqqt1/detection_loss‚2.626144911845525 /loss/FBt4/detection_loss‚0.5464663383544894 trainer/global_step‚5999 .loss/FRt2/detection_loss‚2.277944383566999 2loss/SRqqt3/assignment_loss‚1.2730312979528828 1loss/FBt4/assignment_loss‚0.09060928742181872 /loss/FRt4/detection_loss‚1.5727273811741422 1loss/SRqqt4/detection_loss‚0.8240871750205369 &loss/total_loss‚40.767980217834705 /loss/FRt2/assignment_loss‚2.688538181476104 0loss/FRt1/assignment_loss‚3.4410378463724824 0loss/FRt3/assignment_loss‚2.3338133881731733 2loss/SRqqt1/assignment_loss‚1.9196552587052187 0loss/FBt2/assignment_loss‚0.4143981284162562 2loss/FBt3/assignment_loss‚0.030508069309871644 _runtime‚6360.416025453 epoch‚4]¢Ufˆà 2loss/FBt3/assignment_loss‚0.030508069309871644 _runtime‚6360.416025453 epoch‚4 /loss/FRt3/detection_loss‚2.2568432252581525 1loss/SRqqt2/detection_loss‚2.3600781206041574 _step‚491 0loss/FRt4/assignment_loss‚1.2682880597809951 2loss/SRqqt2/assignment_loss‚1.7680761869996786 1loss/FBt1/assignment_loss‚0.48183159969214884 /loss/FRt1/detection_loss‚2.3617098952861544 /loss/FBt1/detection_loss‚2.8669481916087016 % _timestamp‚1.7884942607758744e+09 /loss/FBt3/detection_loss‚2.6099713921546934 2loss/SRqqt4/assignment_loss‚1.2121739398639697 1loss/SRqqt3/detection_loss‚1.4870544701578126 /loss/FBt2/detection_loss‚2.0560434886387418 0loss/SRqqt1/detection_loss‚2.626144911845525 /loss/FBt4/detection_loss‚0.5464663383544894 trainer/global_step‚5999 .loss/FRt2/detection_loss‚2.277944383566999 2loss/SRqqt3/assignment_loss‚1.2730312979528828 1loss/FBt4/assignment_loss‚0.09060928742181872 /loss/FRt4/detection_loss‚1.5727273811741422 1loss/SRqqt4/detection_loss‚0.8240871750205369 &loss/total_loss‚40.767980217834705 /loss/FRt2/assignment_loss‚2.688538181476104 0loss/FRt1/assignment_loss‚3.4410378463724824 0loss/FRt3/assignment_loss‚2.3338133881731733 2loss/SRqqt1/assignment_loss‚1.9196552587052187 0loss/FBt2/assignment_loss‚0.4143981284162562ë{}"~‰‚(:ó ÃéÔø׆± disk.sda1.out‚0 network.recv‚ 2.10172e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.9140625 disk.sda1.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚155731.7998046875 disk.nvme0n1.in‚ 23621.078125 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚155731.7998046875# disk.dm-0.out‚155731.7998046875* proc.memory.percent‚17.921805381774902 network.sent‚ 1.4546941e+07% disk./.usageGB‚1172.7097396850586 disk.sda3.out‚ 505.19921875 disk.dm-1.in‚0( proc.memory.availableMB‚ 1117.9453125# proc.memory.rssMB‚ 2936.30859375 cpu‚25.23798079345072( disk./.usagePercent‚78.7015345992749 disk.dm-1.out‚ 505.19921875 disk.sda.out‚ 505.19921875" disk.nvme0c0n1.in‚ 23621.078125 disk.dm-0.in‚ 23621.078125$ memory_percent‚93.17660331726074 proc.cpu.threads‚17ì‚òÆ’Š‚(:‡ ÃéÔÎÖÁ¥ gpu.0.gpu‚49.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ƒ}6’‹‚(:‡ ÒƒéÔØɆ­ gpu.0.gpu‚20.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.882' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9764 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žˆç ~Œ‚(:ó ÒƒéÔÕÑß® disk.sda3.out‚ 505.5859375 disk.drbd1260.out‚ 0.9140625 disk.nvme0n1.in‚ 23621.078125 disk.dm-0.in‚ 23621.078125 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 1.4644396e+07) disk./.usagePercent‚78.69370230477234 disk.sda.in‚0 disk.sda.out‚ 505.5859375 disk.sda2.in‚0( disk.nvme0c0n1.out‚155829.2880859375& disk.nvme0n1.out‚155829.2880859375 network.recv‚ 2.107392e+06 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 23621.078125$ memory_percent‚93.17820072174072) proc.memory.availableMB‚ 1117.68359375* proc.memory.percent‚17.921805381774902$ disk./.usageGB‚1172.593032836914 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 505.5859375# disk.dm-0.out‚155829.2880859375# proc.memory.rssMB‚ 2936.30859375 cpu‚25.237602226612438og§?‚(:’ áƒéÔŸš¥± gpu.0.gpu‚49.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.454' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.890800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~«9³|Ž‚(:ñ áƒéÔŠÁ²·" disk.dm-0.out‚156183.107421875 disk.sda.in‚0 disk.sda.out‚ 505.9453125 cpu‚25.23732944059878 proc.cpu.threads‚17 network.recv‚ 2.112212e+06 disk.dm-1.out‚ 505.9453125 disk.sda2.out‚0 disk.dm-0.in‚ 23621.078125$ memory_percent‚93.17824840545654) proc.memory.availableMB‚ 1117.67578125# proc.memory.rssMB‚ 2936.30859375* proc.memory.percent‚17.921805381774902 disk.sda1.out‚0 network.sent‚ 1.4650631e+07% disk./.usageGB‚1173.1827964782715 disk.sda3.out‚ 505.9453125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.91796875 disk.sda1.in‚0 disk.sda2.in‚0) disk./.usagePercent‚78.73328183758849 disk.sda3.in‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 23621.078125' disk.nvme0c0n1.out‚156183.107421875 disk.nvme0n1.in‚ 23621.078125% disk.nvme0n1.out‚156183.10742187536^µ’‚(:‡ ðƒéԸ gpu.0.gpu‚52.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚113.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùÓ<€‚(:õ ðƒéÔÓæ° disk.dm-0.in‚ 23738.1640625' proc.memory.availableMB‚ 1103.828125! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 network.sent‚ 1.4658414e+07 network.recv‚ 2.11635e+06 disk.sda3.out‚ 506.5390625 disk.drbd1260.out‚ 0.91796875 disk.sda2.in‚0! disk.nvme0n1.in‚ 23738.1640625$ memory_percent‚93.26276779174805 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 cpu‚25.249834932524223 disk.sda.out‚ 506.5390625) disk./.usagePercent‚78.74741729359157$ disk./.usageGB‚1173.393424987793 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚156633.79248046875$ disk.dm-0.out‚156633.79248046875 disk.dm-1.out‚ 506.5390625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 23738.1640625' disk.nvme0n1.out‚156633.79248046875&yëZ‘‘‚(:† ÿƒéÔèÜÿ® gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚172.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­¾`Ì_’‚(:Ô ÿƒéÔÿ´î¯ proc.cpu.threads‚17 network.sent‚ 1.4663387e+07) disk./.usagePercent‚78.75741566784133$ disk./.usageGB‚1173.542407989502 disk.dm-1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 23738.1640625$ disk.nvme0n1.out‚156897.83984375 disk.sda3.out‚507 disk.drbd1260.in‚0 disk.sda.out‚507 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 23738.1640625 disk.sda3.in‚0$ memory_percent‚93.25673580169678 cpu‚25.26066358089896 network.recv‚ 2.119326e+06 disk.sda.in‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 2949.640625 disk.dm-1.out‚507 disk.drbd1260.out‚0.921875& disk.nvme0c0n1.out‚156897.83984375! disk.nvme0n1.in‚ 23738.1640625! disk.dm-0.out‚156897.83984375) proc.memory.availableMB‚ 1104.81640625* proc.memory.percent‚18.003177642822266:bCu’“‚(:‡ Ž„éÔƒ´ê¯ gpu.0.gpu‚43.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚172.518' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5036 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0㳌|”‚(:ñ Ž„éÔòÄα disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 507.58203125 disk.drbd1260.in‚0$ memory_percent‚93.25973987579346) disk./.usagePercent‚78.77422742660734# disk.nvme0c0n1.in‚ 23738.1640625& disk.nvme0c0n1.out‚157258.61328125$ disk.nvme0n1.out‚157258.61328125 disk.sda.out‚ 507.58203125 proc.cpu.threads‚17 network.sent‚ 1.4667646e+07% disk./.usageGB‚1173.7929153442383 disk.sda3.out‚ 507.58203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1104.32421875* proc.memory.percent‚18.003177642822266 disk.dm-0.in‚ 23738.1640625! disk.dm-0.out‚157258.61328125 disk.drbd1260.out‚0.921875 disk.sda.in‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2949.640625 cpu‚25.271688936399837 network.recv‚ 2.122584e+06! disk.nvme0n1.in‚ 23738.1640625 ‹>‰2•- _wandbruntime‚6461  _runtime‚6461Þ~íXÄ–ÛT VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 105933848 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28115 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚423020822604288 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚418381792983114 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578072504 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚423020822604288 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6216911176704 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 105933848 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18217 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚94447115 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚412187243718288 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚429237751783936 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18217 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚476196904532992 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _step‚492 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 10929229312 trainer/global_step‚6044 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚191 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚476196749702656 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27846 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚94446924 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚189 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882825 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚412187260757648 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 12155092992 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 240122132 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚429237751783936 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6216911176704 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 11892004352 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18215 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚429237153736704 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚469714986916864 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 6481893676032 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6193954152962 % _timestamp‚1.7884943615560744e+09 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6216331132416 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚423020840607232 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚418381197871250 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 _runtime‚6461.196941601 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _step‚492 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 10929229312 trainer/global_step‚6044 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚191 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚476196749702656 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27846 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚94446924 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚189 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882825 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚412187260757648 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 12155092992 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 240122132 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚429237751783936 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6216911176704 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 11892004352 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18215 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚429237153736704 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚469714986916864 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 6481893676032 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6193954152962 % _timestamp‚1.7884943615560744e+09 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6216331132416 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚423020840607232 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚418381197871250 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 _runtime‚6461.196941601 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 105946818 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6194995386430 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1005 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚429283086484480 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚60748183 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12155092992 _runtime‚6462.264482465 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 580044288 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚476244192596480 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6482358424576 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 240125181 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28115 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 11892004352 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1075 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚469761965062144 trainer/global_step‚6044 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚423066290603520 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚429283684531712 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚94460176 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚33711991 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 105946820 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580044288 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18217 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 240125181 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚476244347426816 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595111864 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 6216795880960 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚33711802 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 240143398 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚429283684531712 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 105946820 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892004352 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 578072504 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚189 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 154830336!6X¼>šéR WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 240143398 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚429283684531712 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 105946820 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892004352 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 578072504 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚189 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 154830336 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12155092992 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 105946818 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6194995386430 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1005 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚429283086484480 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚60748183 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12155092992 _runtime‚6462.264482465 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 580044288 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚476244192596480 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6482358424576 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 240125181 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28115 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 11892004352 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1075 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚469761965062144 trainer/global_step‚6044 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚423066290603520 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚429283684531712 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚94460176 [NDeviceStatsMonitor.on_vaµéïª1lidation_batch_end/inactive_split.small_pool.allocated‚33711991 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 105946820 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580044288 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18217 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 240125181 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚476244347426816 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595111864 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 6216795880960 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚33711802 B&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2013888955116272 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9890800714492798 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.26386234164237976 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.33179014921188354 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.25929728150367737 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.3982163369655609 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11325495690107346 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30121952295303345 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3020716905593872 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.34734535217285156 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.32510286569595337 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4493037164211273 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.44479164481163025 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.3333333432674408 jet/accuracy_5_of_5‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24217918515205383 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2830449044704437 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2738095223903656 +jet/accuracy_2_of_4‚0.36618250608444214 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.456327348947525 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.004649671260267496 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7080075740814209 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989210963249207 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9899387955665588 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05184132233262062 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3929252326488495 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.7777777910232544 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.39497077465057373 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2721189856529236 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.18245601654052734 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9649801850318909 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833432674408 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2966504395008087 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17945460975170135 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4951341152191162 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23087802529335022 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3492870330810547 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17776715755462646 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 particle/accuracy_2_of_7‚1 particle/accuracy_5_of_7‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1190476194024086 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.33517903089523315 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5350741744041443 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9927274584770203 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5428557395935059 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.39794906973838806 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07414234429597855 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10478640347719193 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.7777777910232544 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4583333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969126582145691 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3925911486148834 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4265809953212738 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4131945073604584 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3474312126636505 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4112512767314911 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02372134104371071 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.30404040217399597 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4087301790714264 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.08364121615886688 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36510419845581055 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9586925506591797 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.545192301273346 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025078054517507553 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32314813137054443 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.324961394071579 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9706896543502808 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2756756842136383 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3405172526836395 particle/accuracy_1_of_5‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05109960585832596 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04348365589976311 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2339080423116684 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4417819380760193 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1944444477558136 jet/accuracy_5_of_6‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3470970690250397 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989210963249207 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9897871613502502 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9259259104728699 particle/accuracy_5_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_2_of_6‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4303439259529114 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5717678666114807 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49716562032699585 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3417900800704956 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021516628563404083 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.98859703540802 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10218578577041626 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19922210276126862 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.40594860911369324 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9908230304718018 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3166666626930237 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07304764539003372 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9886974692344666 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.40832221508026123 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 jet/accuracy_3_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3091941773891449 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.32804447412490845 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5723521709442139 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5263158082962036 )jet/accuracy_1_of_3‚0.845005452632904 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969176650047302 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 +jet/accuracy_2_of_3‚0.23551401495933533 *jet/accuracy_2_of_6‚0.5641025304794312 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9226319193840027 (particle/accuracy‚0.8473357558250427 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 ("Purity/3FRt1SRqqt2FBt/event_õî$‰20purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 trainer/global_step‚6044 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.37719711661338806 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 jet/accuracy_2_of_7‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.033147722482681274 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9695596098899841 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.35760918259620667 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6470659375190735 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 particle/accuracy_2_of_3‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3532736897468567 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.41203704476356506 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.349455326795578 epoch‚4 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0017636684933677316 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.27474749088287354 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 6validation_average_jet_accuracy‚0.4004395604133606 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992769360542297 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2630995213985443 5 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29671710729599 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_3_of_4‚0.0635102167725563 jet/accuracy_7_of_7‚0 particle/accuracy_3_of_7‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.809431791305542 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.022002601996064186 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23346379399299622 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 _runtime‚6477.085227599 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2687201201915741 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873681664466858 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02137216553092003 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3341158926486969 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0032051282469183207 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.39087268710136414 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23346379399299622 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3578868806362152 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.26726824045181274 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 *jet/accuracy_1_of_1‚0.4259147346019745 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.30000001192092896 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 $ _timestamp‚1.788494377442762e+09 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.36255326867103577 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14652366936206818 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11225152760744095 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2748936116695404 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6845179796218872 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5706093311309814 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.987528383731842 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4166666567325592 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012629290111362934 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.38226568698883057 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4593023359775543 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9947922229766846 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5204451680183411 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.25946494936943054 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3943445086479187 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5750348567962646 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ( Purity/3FRt4SRqqt0FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.28625351190567017 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03630952164530754 particle/accuracy_4_of_5‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9832133054733276 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3834742605686188 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.558635950088501 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25416678190231323 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ( Purity/3FRt4SRqqt*FBt/FRt_purity‚0.5 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.27409398555755615 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5628864169120789 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_6‚0.9615384340286255 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4690224230289459 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3901515007019043 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02435236983001232 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2851316034793854 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3399936258792877 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.281831830739975 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3463170826435089 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5803571343421936 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,particle/sensitivity‚0.32368049025535583 ,jet/accuracy_3_of_6‚0.057692307978868484 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04528301954269409 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.38851433992385864 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚40.40532307022448 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.28954315185546875 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9831191897392273 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.22758622467517853 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_5‚1 +particle/specificity‚0.9442335963249207 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9855282306671143 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20467837154865265 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.275027871131897 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9314420223236084 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2570207417011261 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ÛòÀOŸ§ ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2738095223903656 +jet/accuracy_2_of_4‚0.36618250608444214 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.456327348947525 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.004649671260267496 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7080075740814209 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989210963249207 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9899387955665588 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05184132233262062 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3929252326488495 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.7777777910232544 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.39497077465057373 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2721189856529236 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.18245601654052734 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9649801850318909 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833432674408 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2966504395008087 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17945460975170135 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4951341152191162 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23087802529335022 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3492870330810547 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17776715755462646 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 particle/accuracy_2_of_7‚1 particle/accuracy_5_of_7‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1190476194024086 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.33517903089523315 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5350741744041443 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9927274584770203 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5428557395935059 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.39794906973838806 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07414234429597855 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10478640347719193 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.7777777910232544 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4583333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969126582145691 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3925911486148834 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4265809953212738 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4131945073604584 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3474312126636505 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4112512767314911 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02372134104371071 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.30404040217399597 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4087301790714264 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.08364121615886688 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36510419845581055 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9586925506591797 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.545192301273346 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025078054517507553 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32314813137054443 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.324961394071579 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9706896543502808 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2756756842136383 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3405172526836395 particle/accuracy_1_of_5‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05109960585832596 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04348365589976311 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2339080423116684 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4417819380760193 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1944444477558136 jet/accuracy_5_of_6‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3470970690250397 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989210963249207 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9897871613502502 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9259259104728699 particle/accuracy_5_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_2_of_6‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4303439259529114 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5717678666114807 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49716562032699585 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3417900800704956 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021516628563404083 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.98859703540802 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10218578577041626 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19922210276126862 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.40594860911369324 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9908230304718018 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3166666626930237 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07304764539003372 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9886974692344666 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.40832221508026123 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 jet/accuracy_3_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3091941773891449 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.32804447412490845 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5723521709442139 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5263158082962036 )jet/accuracy_1_of_3‚0.845005452632904 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969176650047302 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 +jet/accuracy_2_of_3‚0.23551401495933533 *jet/accuracy_2_of_6‚0.5641025304794312 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9226319193840027 (particle/accuracy‚0.8473357558250427 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 trainer/global_step‚6044 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.37719711661338806 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 jet/accuracy_2_of_7‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.033147722482681274 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9695596098899841 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.35760918259620667 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6470659375190735 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 particle/accuracy_2_of_3‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3532736897468567 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.41203704476356506 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.349455326795578 epoch‚4 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0017636684933677316 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.27474749088287354 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 6validation_average_jet_accuracy‚0.4004395604133606 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992769360542297 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2630995213985443 5 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29671710729599 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_3_of_4‚0.0635102167725563 jet/accuracy_7_of_7‚0 particle/accuracy_3_of_7‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.809431791305542 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.022002601996064186 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23346379399299622 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 _runtime‚6477.085227599 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2687201201915741 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873681664466858 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02137216553092003 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3341158926486969 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0032051282469183207 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.39087268710136414 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23346379399299622 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3578868806362152 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.26726824045181274 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 *jet/accuracy_1_of_1‚0.4259147346019745 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.30000001192092896 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 $ _timestamp‚1.788494377442762e+09 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.36255326867103577 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14652366936206818 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11225152760744095 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2748936116695404 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6845179796218872 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5706093311309814 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.987528383731842 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4166666567325592 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012629290111362934 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.38226568698883057 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4593023359775543 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9947922229766846 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5204451680183411 :"Purity/1FRt0SRqqt1+7UÎäCFBt/event_purity‚0.25946494936943054 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3943445086479187 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5750348567962646 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ( Purity/3FRt4SRqqt0FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.28625351190567017 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03630952164530754 particle/accuracy_4_of_5‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9832133054733276 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3834742605686188 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.558635950088501 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25416678190231323 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ( Purity/3FRt4SRqqt*FBt/FRt_purity‚0.5 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.27409398555755615 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5628864169120789 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_6‚0.9615384340286255 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4690224230289459 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3901515007019043 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02435236983001232 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2851316034793854 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3399936258792877 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.281831830739975 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3463170826435089 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5803571343421936 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,particle/sensitivity‚0.32368049025535583 ,jet/accuracy_3_of_6‚0.057692307978868484 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04528301954269409 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.38851433992385864 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚40.40532307022448 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.28954315185546875 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9831191897392273 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.22758622467517853 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_5‚1 +particle/specificity‚0.9442335963249207 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9855282306671143 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20467837154865265 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.275027871131897 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9314420223236084 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2570207417011261 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2950230538845062 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.356535941362381 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2742999196052551 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 jet/accuracy_1_of_7‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,jet/accuracy_4_of_4‚0.006176274735480547 particle/accuracy_2_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473056793213 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2552540898323059 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.514020562171936 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9879782199859619 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5566666722297668 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1524980515241623 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.022373396903276443 particle/accuracy_1_of_4‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.38740673661231995 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 particle/accuracy_3_of_3‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.38271602988243103 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2450958639383316 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ) Purity/*FRt4SRqqt*FBt/FRt_purity‚0.25 jet/accuracy_6_of_6‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.42105263471603394 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2514936625957489 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5806547403335571 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3774985671043396 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16071945428848267 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ) Purity/*FRt4SRqqt0FBt/FRt_purity‚0.25 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9835500121116638 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3417900800704956 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.006896551698446274 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26242268085479736 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.28374966979026794 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30674001574516296 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.27126383781433105 *jet/accuracy_2_of_2‚0.1021895781159401 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 particle/accuracy_1_of_6‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,jet/accuracy_4_of_5‚0.008243728429079056 particle/accuracy_3_of_4‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4167269170284271 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.30092599987983704 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9919830560684204 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24614198505878448 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5059027671813965 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.24999995529651642 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 _step‚494 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2605007290840149 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9703405499458313 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_4_of_6‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.29024508595466614 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.32950183749198914 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9479167461395264 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23346583545207977 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29836326837539673 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9907998442649841 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9967972636222839 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3772827684879303 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 )jet/accuracy_1_of_4‚0.884445071220398 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 particle/accuracy_5_of_5‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9285714030265808 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2271082103252411 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt1SRqqt1FBt/event_purity‚0.010416666977107525 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_5‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.390658438205719 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3178361654281616 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02472415380179882 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4020891785621643 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.48531752824783325 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49568966031074524 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3471429944038391 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9444443583488464 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010465497151017189 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.29844579100608826 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3700190484523773 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9967907667160034 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016688914969563484 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 jet/accuracy_4_of_7‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2013888955116272 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9890800714492798 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.26386234164237976 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.33179014921188354 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.25929728150367737 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.3982163369655609 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11325495690107346 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30121952295303345 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3020716905593872 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.34734535217285156 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.32510286569595337 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4493037164211273 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.44479164481163025 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.3333333432674408 jet/accuracy_5_of_5‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24217918515205383 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2830449044704437î ñx“ ‚(:… ¬„éÔ…ûù° gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6772 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|2/¾z¡‚(:ï ¬„éÔú”Œ³! proc.memory.rssMB‚ 2949.640625( disk./.usagePercent‚78.7649858106288) disk.nvme0c0n1.out‚157578.38037109375' disk.nvme0n1.out‚157578.38037109375 disk.dm-1.out‚ 508.53125 disk.drbd1260.out‚ 0.9296875 disk.sda1.in‚0* proc.memory.percent‚18.003177642822266 proc.cpu.threads‚17 network.sent‚ 1.4780552e+07% disk./.usageGB‚1173.6552085876465 disk.nvme0n1.in‚ 23799.578125 disk.sda2.in‚0( proc.memory.availableMB‚ 1678.5859375 network.recv‚ 2.134198e+06$ disk.dm-0.out‚157578.38037109375 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 508.53125 disk.sda2.out‚0 cpu‚25.293070794293325$ disk.nvme0c0n1.in‚23799.44921875 disk.dm-0.in‚ 23799.890625 disk.sda3.in‚0 disk.sda3.out‚ 508.53125 disk.dm-1.in‚0 disk.sda1.out‚0$ memory_percent‚89.75472450256348?:‰Ó2¢- _wandbruntime‚6486  _runtime‚6486¾ŽAôÆ£À & lr-AdamW/pg1‚0.0004644561809374768 & lr-AdamW/pg2‚0.0004644561809374768 trainer/global_step‚6049 $ _timestamp‚1.788494386925947e+09 _runtime‚6486.566583738 _step‚495Øš;ñˤŠ& lr-AdamW/pg2‚0.0004644561809374768 trainer/global_step‚6049 $ _timestamp‚1.788494386925947e+09 _runtime‚6486.566583738 _step‚495 & lr-AdamW/pg1‚0.0004644561809374768ïÄûãÉ2¥- _wandbruntime‚6486  _runtime‚6486Ÿ¾-üø'¦òO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423928174350848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6492545579008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423928192353792 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106182952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚413043651222672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚470674098262016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚470674074321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 240554788 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106182950 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134394854 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33797346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚6049 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134371838 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134394854 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423928192353792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423928174350848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚493 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232762368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6205025941032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240577806 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚430154844764672 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94687370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60890517 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6204255292620 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106182950 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6227443058688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚419248694203064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208822272 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚413043668262032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6492754401280 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚430155635412480 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚430155635412480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33796855 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106182952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016–VGÐæ §÷O QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240577806 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚430154844764672 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94687370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60890517 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 OFDeviceStatsMonitor.on_train_batch_start/inactÄx{Oive_split.small_pool.peak‚1046 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6204255292620 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106182950 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6227443058688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚419248694203064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208822272 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚413043668262032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6492754401280 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚430155635412480 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚430155635412480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33796855 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106182952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423928174350848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6492545579008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423928192353792 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106182952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚413043651222672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚470674098262016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚470674074321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 240554788 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106182950 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134394854 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33797346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚6049 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134371838 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134394854 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423928192353792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423928174350848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚493 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232762368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6205025941032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ð–Ï¥2¨- _wandbruntime‚6487  _runtime‚6487ž)kë'©ýM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94702123 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6493783147520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33801960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚419314974090930 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240616417 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚430224340110848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240616417 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106199824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _runtime‚6487.514910981 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134416591 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚430224340110848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423995874290176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94702624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚413109734697792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423995892293120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚430223549449728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106199826 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6493582727168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚470749446604800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470749422664704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚413109717658432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚423995874290176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134416591 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚497 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134393549 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2´FëÇ'ª‚N >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚497 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134393549 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94702123 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6493783147520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33801960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚419314974090930 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240616417 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚430224340110848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240616417 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106199824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _runtime‚6487.514910981 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134416591 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚430224340110848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423995874290176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94702624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚413109734697792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423995892293120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚430223549449728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106199826 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6493582727168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚470749446604800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470749422664704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚413109717658432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚423995874290176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134416591ñâsVï2«- _wandbruntime‚6487  _runtime‚6487Ïi ÷[¬Õ /loss/FRt3/assignment_loss‚2.663842196367225 2loss/SRqqt3/assignment_loss‚1.2441125409677625 1loss/FBt3/assignment_loss‚0.19231815815631612 epoch‚5 _runtime‚6487.517076921 0loss/FBt1/assignment_loss‚0.2673477845394039 1loss/FBt4/assignment_loss‚0.16895879909432956 1loss/SRqqt1/detection_loss‚2.6793216986040913 1loss/SRqqt2/detection_loss‚2.6824009522147803 /loss/FBt3/detection_loss‚2.4820229709148407 trainer/global_step‚6049 0loss/FRt1/assignment_loss‚2.9683591609305524 /loss/FRt2/assignment_loss‚3.474374200616564 /loss/FRt2/detection_loss‚2.9123319877218106 /loss/FBt1/detection_loss‚2.3212437629699707 0loss/FRt4/assignment_loss‚1.4885938061135155 2loss/SRqqt2/assignment_loss‚1.4744566087813482 2loss/SRqqt4/assignment_loss‚0.8356510631310252 /loss/FBt4/detection_loss‚0.7598028793895187 2loss/SRqqt1/assignment_loss‚1.6461268314911472 1loss/SRqqt3/detection_loss‚1.6313832852869266 /loss/FRt1/detection_loss‚2.3147459087219646 0loss/SRqqt4/detection_loss‚1.196850667349421 /loss/FBt2/assignment_loss‚0.522707406224476 .loss/FRt3/detection_loss‚2.225485217814543 .loss/FBt2/detection_loss‚2.074046678013272 % _timestamp‚1.7884943878768275e+09 /loss/FRt4/detection_loss‚1.8211716627702117 %loss/total_loss‚42.04765622818502 _step‚498k”2×`­Ú 0loss/FBt1/assignment_loss‚0.2673477845394039 1loss/FBt4/assignment_loss‚0.16895879909432956 1loss/SRqqt1/detection_loss‚2.6793216986040913 1loss/SRqqt2/detection_loss‚2.6824009522147803 /loss/FBt3/detection_loss‚2.4820229709148407 trainer/global_step‚6049 0loss/FRt1/assignment_loss‚2.9683591609305524 /loss/FRt2/assignment_loss‚3.474374200616564 /loss/FRt2/detection_loss‚2.9123319877218106 /loss/FBt1/detection_loss‚2.3212437629699707 0loss/FRt4/assignment_loss‚1.4885938061135155 2loss/SRqqt2/assignment_loss‚1.4744566087813482 2loss/SRqqt4/assignment_loss‚0.8356510631310252 /loss/FBt4/detection_loss‚0.7598028793895187 2loss/SRqqt1/assignment_loss‚1.6461268314911472 1loss/SRqqt3/detection_loss‚1.6313832852869266 /loss/FRt1/detection_loss‚2.3147459087219646 0loss/SRqqt4/detection_loss‚1.196850667349421 /loss/FBt2/assignment_loss‚0.522707406224476 .loss/FRt3/detection_loss‚2.225485217814543 .loss/FBt2/detection_loss‚2.074046678013272 % _timestamp‚1.7884943878768275e+09 /loss/FRt4/detection_loss‚1.8211716627702117 %loss/total_loss‚42.04765622818502 _step‚498 /loss/FRt3/assignment_loss‚2.663842196367225 2loss/SRqqt3/assignment_loss‚1.2441125409677625 1loss/FBt3/assignment_loss‚0.19231815815631612 epoch‚5 _runtime‚6487.517076921òüL„®‚(:’ »„éÔ–¼Þ± gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚174.305' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.861000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žé£íÞå°‚(:… Ê„éÔ®ùÒ¯ gpu.0.gpu‚14.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.53' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.706 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qi(”v±‚(:ë Ê„éÔÖ÷ß³ cpu‚25.29158035289467% disk./.usageGB‚1173.8938217163086 disk.dm-1.out‚509.4375 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 1.5012721e+07) disk./.usagePercent‚78.78099933790308" disk.nvme0c0n1.in‚ 23861.015625) disk.nvme0c0n1.out‚159718.71142578125' disk.nvme0n1.out‚159718.71142578125 disk.drbd1260.out‚ 0.93359375 disk.sda.in‚0 disk.sda.out‚509.4375 network.recv‚ 2.145721e+06 disk.dm-0.in‚ 23861.015625 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# memory_percent‚89.7533655166626! proc.memory.rssMB‚ 2949.640625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 23861.015625$ disk.dm-0.out‚159718.71142578125 disk.sda3.out‚509.4375 disk.drbd1260.in‚0) proc.memory.availableMB‚ 1678.80859375* proc.memory.percent‚18.003177642822266¤¢%òc²‚(:‡ Ù„éÔõ¶ ± gpu.0.gpu‚33.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.286' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryL ‰/Errors‚0# gpu.0.uncorrectedMemoryErrors‚0Å×wGr³‚(:ç Ù„éÔ¨Ëð´& disk.nvme0n1.out‚160047.1650390625 disk.dm-0.in‚ 23861.015625 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0) disk./.usagePercent‚78.80203726904433 disk.dm-1.out‚ 509.890625 disk.sda.in‚0 disk.sda.out‚ 509.890625 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 23861.015625$ memory_percent‚89.75334167480469! proc.memory.rssMB‚ 2949.640625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.9375# disk.dm-0.out‚160047.1650390625 disk.sda3.out‚ 509.890625% proc.memory.availableMB‚ 1678.8125 proc.cpu.threads‚17 network.recv‚ 2.150435e+06 disk.dm-1.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚160047.1650390625* proc.memory.percent‚18.003177642822266 cpu‚25.291098345687093 network.sent‚ 1.501897e+07% disk./.usageGB‚1174.2073020935059 disk.nvme0n1.in‚ 23861.015625Töu2´- _wandbruntime‚6536  _runtime‚6536Ë%WǵÁ & lr-AdamW/pg1‚0.0004588165305467073 & lr-AdamW/pg2‚0.0004588165305467073 trainer/global_step‚6099 % _timestamp‚1.7884944366604316e+09 _runtime‚6536.300886018 _step‚499®Û– Ì¶Æ & lr-AdamW/pg1‚0.0004588165305467073 & lr-AdamW/pg2‚0.0004588165305467073 trainer/global_step‚6099 % _timestamp‚1.7884944366604316e+09 _runtime‚6536.300886018 _step‚499ów#o?2·- _wandbruntime‚6536  _runtime‚6536SÂ¥ú'¸ôO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6276922114048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚433590093431296 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34058890 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95431805 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61372915 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚433590884079104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6277694758912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135483756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95432329 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242487390 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135460740 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚427313171317248 _runtime‚6536.302079677 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107026650 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242510408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480986361383936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6255096526140 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚427313189320192 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚433590884079104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213888000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6276922114048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚422602086576172 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6545059870720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427313189320192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61372917 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34059412 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6544869922816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135483756 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚416346973010672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135460740 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚433590093431296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016<èÍ<ÿ'¹ùO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95432329 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242487390 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135460740 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚427313171317248 _runtime‚6536.302079677 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107026650 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242510408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480986361383936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6255096526140 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚427313189320192 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚433590884079104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213888000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6276922114048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚422602086576172 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6545059870720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427313189320192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61372917 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34059412 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6544869922816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135483756 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚416346973010672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135460740 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚433590093431296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6276922114048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚433590093431296 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34058890 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95431805 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61372915 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚433590884079104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6277694758912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135483756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328ô%ÜCS2º- _wandbruntime‚6537  _runtime‚6537½W“å'»þM ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚481062762904576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135482451 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚433659588777472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6546115132416 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚474516839803904 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6256097666226 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚422669154152018 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107043526 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427380871256576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚422668366464038 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107043524 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚433659588777472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242549019 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242525975 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34064732 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 % _timestamp‚1.7884944377651045e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135482451 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427380889259520 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6277926859776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚433658798116352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6255327017606 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416413056485792 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚481062978876416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 _step‚501 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242549019 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34064215 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6278699517952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6278699517952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61382563 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427380889259520 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135505493 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚474516863744000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416413039446432 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 _runtime‚6537.406041607 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95446778 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242549019 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H:DeviceStatsMonitor.on_train_batch_end/alloe®Í+$cation.all.freed‚ 242525975 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34064732 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 % _timestamp‚1.7884944377651045e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135482451 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427380889259520 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6277926859776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚433658798116352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6255327017606 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416413056485792 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚481062978876416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 _step‚501 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242549019 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34064215 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6278699517952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6278699517952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61382563 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427380889259520 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135505493 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚474516863744000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416413039446432 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 _runtime‚6537.406041607 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95446778 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135482451 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚433659588777472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6546115132416 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚474516839803904 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6256097666226 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚422669154152018 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107043526 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427380871256576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚422668366464038 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107043524õÞ)I2½- _wandbruntime‚6537  _runtime‚6537ºAdÈ[¾Õ 1loss/FBt2/assignment_loss‚0.22188706470496983 /loss/FRt4/detection_loss‚1.7372807983730763 -loss/FBt1/detection_loss‚2.51399547457695 /loss/FBt3/detection_loss‚3.1588890850543976 0loss/FRt1/assignment_loss‚3.1909105282909467 -loss/FRt4/assignment_loss‚1.3680731924075 2loss/FBt3/assignment_loss‚0.006091617826314177 /loss/FRt2/detection_loss‚2.5540643490302926 epoch‚5 1loss/FBt4/assignment_loss‚0.29319022864110844 -loss/FBt2/detection_loss‚2.64955912232399 /loss/FBt4/detection_loss‚0.6988392451021355 _runtime‚6537.407920116 2loss/SRqqt2/assignment_loss‚1.9635827446525747 2loss/SRqqt3/assignment_loss‚1.1111058986468896 1loss/FBt1/assignment_loss‚0.22751380809286276 /loss/FRt3/detection_loss‚2.3347431436181068 trainer/global_step‚6099 2loss/SRqqt1/assignment_loss‚1.0454431755532478 _step‚502 /loss/FRt1/detection_loss‚2.1109812147212477 1loss/SRqqt1/detection_loss‚2.2479828678328415 1loss/SRqqt3/detection_loss‚1.1521570405645951 1loss/SRqqt4/detection_loss‚1.0339520004088596 0loss/FRt2/assignment_loss‚3.7280978163083396 0loss/FRt3/assignment_loss‚1.8379978701472282 /loss/SRqqt2/detection_loss‚2.80615050684322 %loss/total_loss‚40.90970701039291 % _timestamp‚1.7884944377677588e+09 2loss/SRqqt4/assignment_loss‚0.9172182166712137Ñ‘`¿Ú 2loss/SRqqt2/assignment_loss‚1.9635827446525747 2loss/SRqqt3/assignment_loss‚1.1111058986468896 1loss/FBt1/assignment_loss‚0.22751380809286276 /loss/FRt3/detection_loss‚2.3347431436181068 trainer/global_step‚6099 2loss/SRqqt1/assignment_loss‚1.0454431755532478 _step‚502 /loss/FRt1/detection_loss‚2.1109812147212477 1loss/SRqqt1/detection_loss‚2.2479828678328415 1loss/SRqqt3/detection_loss‚1.1521570405645951 1loss/SRqqt4/detection_loss‚1.0339520004088596 0loss/FRt2/assignment_loss‚3.7280978163083396 0loss/FRt3/assignment_loss‚1.8379978701472282 /loss/SRqqt2/detection_loss‚2.80615050684322 %loss/total_loss‚40.90970701039291 % _timestamp‚1.7884944377677588e+09 2loss/SRqqt4/assignment_loss‚0.9172182166712137 1loss/FBt2/assignment_loss‚0.22188706470496983 /loss/FRt4/detection_loss‚1.7372807983730763 -loss/FBt1/detection_loss‚2.51399547457695 /loss/FBt3/detection_loss‚3.1588890850543976 0loss/FRt1/assignment_loss‚3.1909105282909467 -loss/FRt4/assignment_loss‚1.3680731924075 2loss/FBt3/assignment_loss‚0.006091617826314177 /loss/FRt2/detection_loss‚2.5540643490302926 epoch‚5 1loss/FBt4/assignment_loss‚0.29319022864110844 -loss/FBt2/detection_loss‚2.64955912232399 /loss/FBt4/detection_loss‚0.6988392451021355 _runtime‚6537.407920116ö4e^P’À‚(:‡ è„éÔÄ݉­ gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.041' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{Ä]ã|Á‚(:ñ è„éÔ¾‡Ð® disk.nvme0n1.in‚ 23861.015625 disk.dm-0.in‚ 23861.015625# disk.dm-0.out‚160383.9150390625) disk./.usagePercent‚78.81197599353862 disk.dm-1.out‚ 510.62109375 disk.drbd1260.out‚0.9375 disk.sda.in‚0 disk.sda.out‚ 510.62109375 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚160383.9150390625 network.sent‚ 1.5024757e+07 disk.drbd1260.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚160383.9150390625 disk.sda3.out‚ 510.62109375$ memory_percent‚89.75651264190674) proc.memory.availableMB‚ 1678.29296875* proc.memory.percent‚18.003177642822266$ disk./.usageGB‚1174.355396270752 disk.dm-1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 23861.015625 disk.sda3.in‚0! proc.memory.rssMB‚ 2949.640625 cpu‚25.290676915526575 proc.cpu.threads‚17 network.recv‚ 2.154477e+06ù:ŠB‚(:’ ÷„éÔÄØ¥­ gpu.0.gpu‚25.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.384' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.876799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ÓÂ(:ô ÷„éÔ¾ð² disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 24002.5703125 disk.dm-1.out‚ 510.984375 disk.drbd1260.in‚0( proc.memory.availableMB‚ 1119.6171875* proc.memory.percent‚18.003177642822266 disk.sda.in‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚160640.53955078125' disk.nvme0n1.out‚160640.53955078125$ disk.dm-0.out‚160640.53955078125# memory_percent‚93.1663990020752! proc.memory.rssMB‚ 2949.640625 network.sent‚ 1.5132955e+07% disk./.usageGB‚1174.3532943725586 disk.sda.out‚ 510.984375 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 24002.5703125 disk.drbd1260.out‚ 0.94140625 cpu‚25.290255972116626 network.recv‚ 2.160646e+06) disk./.usagePercent‚78.81183493338725! disk.nvme0n1.in‚ 24002.5703125 disk.sda3.in‚0 disk.sda3.out‚ 510.984375½£ï¬‘Ä‚(:† †…éÔŠ¬é° gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.895' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.579 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š¾«u‚Å‚(:÷ †…éÔ¢œ‹µ disk.sda1.in‚0 network.sent‚ 1.5137967e+07 network.recv‚ 2.163971e+06 disk.sda3.in‚0 disk.sda3.out‚ 511.5859375 disk.dm-1.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 24002.5703125) disk./.usagePercent‚78.81183698144753 disk.sda1.out‚0 disk.sda2.out‚0' disk.nvme0n1.out‚160970.04345703125 disk.dm-0.in‚ 24002.5703125! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 proc.cpu.threads‚17 disk.dm-1.out‚ 511.5859375 disk.sda.out‚ 511.5859375 disk.sda2.in‚0# memory_percent‚93.1633472442627( proc.memory.availableMB‚ 1120.1171875 cpu‚25.289878629258833% disk./.usageGB‚1174.3533248901367 disk.drbd1260.out‚ 0.94140625 disk.sda.in‚0# disk.nvme0c0n1.in‚ 24002.5703125) disk.nvme0c0n1.out‚160970.04345703125$ disk.dm-0.out‚160970.04345703125äþþ4Æ‚(:’ •…éÔ¦çÉ° gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.562599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î†xÇ‚(:í •…éÔʤŸµ) disk./.usagePercent‚78.86079765434837% disk./.usageGB‚1175.0828742980957 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 511.90625 disk.sda1.in‚0 disk.sda3.out‚ 511.90625 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚161234.0302734375& disk.nvme0n1.out‚161234.0302734375 disk.dm-0.in‚ 24002.5703125' proc.memory.availableMB‚ 1120.109375 disk.sda3.in‚0 disk.dm-1.out‚ 511.90625 disk.drbd1260.out‚ 0.9453125# disk.nvme0c0n1.in‚ 24002.5703125# disk.dm-0.out‚161234.0302734375$ memory_percent‚93.16339492797852 proc.cpu.threads‚17 disk.sda2.in‚0! disk.nvme0n1.in‚ 24002.5703125! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 cpu‚25.289413173916582 network.sent‚ 1.5144239e+07 network.recv‚ 2.168734e+06íÏGV2È- _wandbruntime‚6587  _runtime‚6587<#OÉÉà ' lr-AdamW/pg2‚0.00045316381451262203 trainer/global_step‚6149 % _timestamp‚1.7884944877220385e+09 _runtime‚6587.362513704 _step‚503 ' lr-AdamW/pg1‚0.00045316381451262203bœgÄÎÊÈ _runtime‚6587.362513704 _step‚503 ' lr-AdamW/pg1‚0.00045316381451262203 ' lr-AdamW/pg2‚0.00045316381451262203 trainer/global_step‚6149 % _timestamp‚1.7884944877220385e+09÷™¸Þ2Ë- _wandbruntime‚6587  _runtime‚6587zÿ ù'ÌóO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437025342097920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96177474 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚484806070425600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6304396462836 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚419650294798672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚484806288507904 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136572658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425954691261508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244419992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚430698186286592 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61855317 _runtime‚6587.364014282 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107870350 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚430698168283648 _step‚504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚430698186286592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6327173814272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437025342097920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107870350 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚419650311838032 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328x …‰2ÍøO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6327946459136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478208908600320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107870352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚437026132745728 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6597355967488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136572658 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚419650294798672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚484806288507904 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136572658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425954691261508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244419992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚430698186286592 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61855317 _runtime‚6587.364014282 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107870350 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚430698168283648 _step‚504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚430698186286592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6327173814272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437025342097920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107870350 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚419650311838032 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437025342097920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96177474 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚484806070425600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6304396462836 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107887226 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478284280883200 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6598189285376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6306168251334 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136571353 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6598402288640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6328178560000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96192990 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96192443 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚430765886225920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136594395 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 _runtime‚6588.427186874 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244481621 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6328951218176 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚430765886225920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34328025 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚430765868222976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61864963 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884944887858768e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419716378273792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚426022546525126 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6305397602714 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚6149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244481621Vïѧ 'ЃN [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚430765886225920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136594395 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 _runtime‚6588.427186874 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244481621 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6328951218176 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚430765886225920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34328025 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚430765868222976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61864963 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884944887858768e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419716378273792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚426022546525126 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6305397602714 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚6149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244481621 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244458577 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107887224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚484882683171840 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚437094837444096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6328178560000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚437094046782976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚419716361234432 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107887226 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478284280883200 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6598189285376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6306168251334 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136571353 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6598402288640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6328178560000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96192990 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96192443 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0ù-DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚481976325739520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚434083165250048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6377425514496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62337717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6378198159360 trainer/global_step‚6199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246352594 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440460590764544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34586163 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚422953633626032 % _timestamp‚1.7884945375364335e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108714052 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚434083183252992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62337715 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137638544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6649560556032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚440461381412352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6377425514496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429308871322388 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚434083183252992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96923878 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 _step‚508 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108714050 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352õÙ[7þ'ßøO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6378198159360 trainer/global_step‚6199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246352594 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440460590764544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34586163 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚422953633626032 % _timestamp‚1.7884945375364335e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108714052 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚434083183252992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62337715 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137638544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6649560556032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚440461381412352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6377425514496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429308871322388 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚434083183252992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96923878 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 _step‚508 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108714050 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137661560 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34586705 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183656448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6354467047944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚488625702639104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚422953616586672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 _runtime‚ 6637.17686538 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108714050 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137638544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚481976325739520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚434083165250048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6377425514496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62337717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489ü¿0¨w2à- _wandbruntime‚6638  _runtime‚6638°4Óž'áûM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440530086110720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137683297 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137660255 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246414223 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246391179 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚488702317047296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 _step‚509 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6379202918400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6650422798848 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429375938898234 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246391179 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482051698022400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6378430260224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34591948 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚429375151210254 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚434150883192320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6378430260224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚440529295449600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚423019700061792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6650619024896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108730926 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62347363 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚434150883192320 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137683297 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚423019683022432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚440530086110720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108730924 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6379202918400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246414223 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96938770 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137660255 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62347365 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 trainer/global_step‚6199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _runtime‚ 6638.0111349 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚434150865189376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚488702096881152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6356238836442 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6355468187822 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992¢Êá½éâ€N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 _step‚509 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6379202918400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6650422798848 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429375938898234 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246391179 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482051698022400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6378430260224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34591948 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚429375151210254 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚434150883192320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6378430260224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 78![³½7687980 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚440529295449600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚423019700061792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6650619024896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108730926 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62347363 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚434150883192320 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137683297 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚423019683022432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚440530086110720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108730924 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6379202918400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246414223 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96938770 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137660255 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62347365 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 trainer/global_step‚6199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _runtime‚ 6638.0111349 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚434150865189376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚488702096881152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6356238836442 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6355468187822 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440530086110720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137683297 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137660255 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246414223 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246391179 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚488702317047296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075ýËG1=2ã- _wandbruntime‚6638  _runtime‚6638õð­’ZäÔ 0loss/FRt4/assignment_loss‚0.8700320370613582 1loss/FBt2/assignment_loss‚0.07352570583609325 % _timestamp‚1.7884945383725781e+09 2loss/SRqqt2/assignment_loss‚1.6760062102613777 trainer/global_step‚6199 _step‚510 1loss/SRqqt3/assignment_loss‚1.635634034452304 1loss/FBt3/assignment_loss‚0.12535837053519572 1loss/SRqqt1/detection_loss‚2.1512293106033686 epoch‚5 1loss/SRqqt3/detection_loss‚1.2347098031199315 .loss/FBt2/detection_loss‚2.191245827409956 _runtime‚6638.012726629 0loss/FRt1/assignment_loss‚2.3877723067998886 1loss/SRqqt1/assignment_loss‚1.776895251169446 1loss/FBt1/assignment_loss‚0.22690066240841134 /loss/FBt3/detection_loss‚2.7349268198013306 0loss/FRt2/assignment_loss‚3.3273706674575805 .loss/FRt3/assignment_loss‚2.48626530300016 2loss/SRqqt4/assignment_loss‚0.8772437307620193 0loss/SRqqt2/detection_loss‚2.307701723328952 &loss/total_loss‚39.358717027684456 -loss/FRt1/detection_loss‚2.78916840814054 /loss/FRt2/detection_loss‚2.3702010045684485 .loss/FBt1/detection_loss‚2.830621153116226 1loss/FBt4/assignment_loss‚0.18214774602256156 .loss/FRt3/detection_loss‚2.245591195705144 /loss/FRt4/detection_loss‚1.5446856290929847 1loss/SRqqt4/detection_loss‚0.7638643138201499 /loss/FBt4/detection_loss‚0.5496198132110294áÄŸÿ_åÙ 1loss/SRqqt3/detection_loss‚1.2347098031199315 .loss/FBt2/detection_loss‚2.191245827409956 _runtime‚6638.012726629 0loss/FRt1/assignment_loss‚2.3877723067998886 1loss/SRqqt1/assignment_loss‚1.776895251169446 1loss/FBt1/assignment_loss‚0.22690066240841134 /loss/FBt3/detection_loss‚2.7349268198013306 0loss/FRt2/assignment_loss‚3.3273706674575805 .loss/FRt3/assignment_loss‚2.48626530300016 2loss/SRqqt4/assignment_loss‚0.8772437307620193 0loss/SRqqt2/detection_loss‚2.307701723328952 &loss/total_loss‚39.358717027684456 -loss/FRt1/detection_loss‚2.78916840814054 /loss/FRt2/detection_loss‚2.3702010045684485 .loss/FBt1/detection_loss‚2.830621153116226 1loss/FBt4/assignment_loss‚0.18214774602256156 .loss/FRt3/detection_loss‚2.245591195705144 /loss/FRt4/detection_loss‚1.5446856290929847 1loss/SRqqt4/detection_loss‚0.7638643138201499 /loss/FBt4/detection_loss‚0.5496198132110294 0loss/FRt4/assignment_loss‚0.8700320370613582 1loss/FBt2/assignment_loss‚0.07352570583609325 % _timestamp‚1.7884945383725781e+09 2loss/SRqqt2/assignment_loss‚1.6760062102613777 trainer/global_step‚6199 _step‚510 1loss/SRqqt3/assignment_loss‚1.635634034452304 1loss/FBt3/assignment_loss‚0.12535837053519572 1loss/SRqqt1/detection_loss‚2.1512293106033686 epoch‚5þ«¾H’æ‚(:‡ Ñ…éÔ»Ïþ­ gpu.0.gpu‚48.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.971' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ3Jž€ç‚(:õ Ñ…éÔƒ¢± network.recv‚ 2.187406e+06( disk.nvme0c0n1.out‚162573.0810546875 disk.sda.in‚0 disk.sda.out‚ 513.6015625 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚93.16396713256836! proc.memory.rssMB‚ 2949.640625" disk.nvme0n1.in‚24137.96484375' proc.memory.availableMB‚ 1120.015625* proc.memory.percent‚18.003177642822266 cpu‚25.287829055436784 proc.cpu.threads‚17% disk./.usageGB‚1175.0818061828613$ disk.nvme0c0n1.in‚24137.96484375# disk.dm-0.out‚162573.0810546875 disk.sda3.out‚ 513.6015625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 network.sent‚ 1.5283008e+07) disk./.usagePercent‚78.86072597223878& disk.nvme0n1.out‚162573.0810546875 disk.dm-0.in‚24137.96484375 disk.sda3.in‚0 disk.dm-1.out‚ 513.6015625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.953125¾¥’è‚(:‡ à…éÔÜ»—¯ gpu.0.gpu‚39.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%ø4>zé‚(:ï à…éÔ“½Õ³ disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚24137.96484375 disk.sda3.out‚ 514.03125 disk.dm-1.out‚ 514.03125 disk.drbd1260.out‚0.953125( proc.memory.availableMB‚ 1120.0078125! proc.memory.rssMB‚ 2949.640625% disk./.usageGB‚1175.5838470458984 disk.dm-0.in‚24137.96484375 disk.sda3.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.89441835579754 disk.sda2.out‚0( disk.nvme0c0n1.out‚162957.8935546875" disk.nvme0n1.in‚24137.96484375# disk.dm-0.out‚162957.8935546875$ memory_percent‚93.16401481628418* proc.memory.percent‚18.003177642822266 cpu‚25.28742854879024 network.sent‚ 1.5380813e+07 network.recv‚ 2.193916e+06 disk.sda.in‚0 disk.sda.out‚ 514.03125 disk.sda1.out‚0& disk.nvme0n1.out‚162957.8935546875O¡œê‚(:‘ ï…éÔ³‡² gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.562599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$%„ë‚(:ù ï…éÔÝß¾¹! proc.memory.rssMB‚ 2949.640625 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.out‚ 514.54296875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.95703125 disk.sda.out‚ 514.54296875) disk.nvme0c0n1.out‚163382.70654296875% disk./.usageGB‚1175.5818138122559 disk.dm-1.in‚0 disk.sda.in‚0$ disk.dm-0.out‚163382.70654296875( proc.memory.availableMB‚ 1119.5234375* proc.memory.percent‚18.003177642822266 cpu‚25.28704135957743 disk.sda1.out‚0 network.recv‚ 2.197642e+06 disk.sda3.out‚ 514.54296875 disk.sda1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 24250.5078125! disk.nvme0n1.in‚ 24250.5078125 disk.dm-0.in‚ 24250.5078125 network.sent‚ 1.5384041e+07( disk./.usagePercent‚78.8942819037818 disk.sda2.in‚0' disk.nvme0n1.out‚163382.70654296875$ memory_percent‚93.16697120666504ßäÅ2ì- _wandbruntime‚6686  _runtime‚6686¿×ðÆíÀ & lr-AdamW/pg1‚0.0004418239000620634 & lr-AdamW/pg2‚0.0004418239000620634 trainer/global_step‚6249 $ _timestamp‚1.788494586686661e+09 _runtime‚6686.327054631 _step‚511‘ËîÅ _step‚511 & lr-AdamW/pg1‚0.0004418239000620634 & lr-AdamW/pg2‚0.0004418239000620634 trainer/global_step‚6249 $ _timestamp‚1.788494586686661e+09 _runtime‚6686.327054631ÿd¨}2ï- _wandbruntime‚6686  _runtime‚6686ßÙÃøú'ðôO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6427677214720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97670506 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6404537633052 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6427677214720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62820117 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚492445514605056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6405308281464 _step‚512 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6428449859584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443895839431168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443896630078976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485743766818816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6701747786240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97671041 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚485743742878720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚426256955414032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248285196 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚6686.328247368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443895839431168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚432662263695496 trainer/global_step‚6249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109557750 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚437468180219392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 % _timestamp‚1.7884945866878014e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6701553644032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443896630078976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6428449859584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚437468162216448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚492445296522752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚426256938374672 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138727446 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825%7ŠJ[ñùO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6427677214720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97670506 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6404537633052 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6427677214720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62820117 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚492445514605056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6405308281464 _step‚512 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6428449859584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443895839431168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443896630078976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485743766818816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6701747786240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97671041 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.largÈR‡¤ e_pool.freed‚485743742878720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚426256955414032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248285196 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚6686.328247368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443895839431168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚432662263695496 trainer/global_step‚6249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109557750 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚437468180219392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 % _timestamp‚1.7884945866878014e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6701553644032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443896630078976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6428449859584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚437468162216448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚492445296522752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚426256938374672 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138727446 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0€K‹î{2ò- _wandbruntime‚6687  _runtime‚6687œ­î'óþM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚443964544116224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚485819091221504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138749157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97685943 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248346825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 _runtime‚6687.300975298 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚432729331271342 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚426323004810432 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚485819115161600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 248323781 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109574626 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492521688708608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6702597487104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492521915166208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138772199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138772199 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 248323781 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97685397 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 _step‚513 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚437535880158720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62829765 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34856178 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6406309421550 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚437535862155776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044BK­¨ 'ôƒN XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6405538772930 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248346825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 _runtime‚6687.300975298 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚432729331271342 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚426323004810432 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚485819115161600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 248323781 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109574626 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492521688708608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6702597487104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492521915166208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138772199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138772199 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 248323781 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97685397 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 _step‚513 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚437535880158720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62829765 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34856178 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6406309421550 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚437535862155776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚443964544116224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚485819091221504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138749157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97685943 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139816348 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6455378866572 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110401452 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6454608218160 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚496264890472960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250240816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63302515 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35115154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚429560260162672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6478701559808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98417669 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110401450 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6478701559808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚429560277202032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚440853177185792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚440853159182848 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6477928914944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚6299 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447331088097792 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884946355079463e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139839364 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚436014868380832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6477928914944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚496265116943872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664c+Iÿ'…ùO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63302515 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35115154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚429560260162672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6478701559808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98417669 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110401450 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6478701559808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚429560277202032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚440853177185792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚440853159182848 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6477928914944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚6299 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447331088097792 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884946355079463e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139839364 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚436014868380832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6477928914944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚496265116943872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139816348 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚440853177185792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139839364 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139816348 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6455378866572 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110401452 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6454608218160 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚496264890472960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250240816 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139861101 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250279427 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63312165 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚429626343637792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6478933660672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚489586532300800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6479706318848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 trainer/global_step‚6299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚440920859122176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489586508360704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚517 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚440920859122176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6736.003005321 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚436082723644450 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110418324 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447399792782848 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110418326 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63312163 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚496341275097600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6479706318848 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250279427 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228554752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250256383 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447400583443968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35120956 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚429626326598432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6456380006658 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98432566 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6754766736896 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139838059 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚496341503652352 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98433121 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚436081935956470 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440920877125120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250256383 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139838059 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35120403 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6754971351552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328/Ü̽ë ˆƒN RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 % _timestamp‚1.7884946363619263e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6455609358038 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚447400583443968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚440920877125120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250279427 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228554752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250256383 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447400583443968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35120956 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚429626326598432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6456380006658 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98432566 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6754766736896 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139838059 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚496341503652352 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98433121 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚436081935956470 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440920877125120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250256383 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139838059 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35120403 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6754971351552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139861101 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250279427 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63312165 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚429626343637792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6478933660672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚489586532300800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6479706318848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 trainer/global_step‚6299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚440920859122176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489586508360704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚517 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚440920859122176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6736.003005321 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚436082723644450 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110418324 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447399792782848 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110418326 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63312163 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚496341275097600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6479706318848 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚444238174152192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252173418 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚439368260753940 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6505449451680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99165728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35380251 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚493278601097216 _runtime‚6785.507443748 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6528953260032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500084533484544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚6349 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚450766336764416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6528180615168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140905250 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 $ _timestamp‚1.788494685866778e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚450767127412224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚450767127412224 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚444238174152192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252173418 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚439368260753940 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6505449451680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99165728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35380251 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚493278601097216 _runtime‚6785.507443748 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6528953260032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500084533484544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStata+À–ÇsMonitor.on_train_batch_start/active.small_pool.peak‚27846 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚6349 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚450766336764416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6528180615168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140905250 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 $ _timestamp‚1.788494685866778e+09 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6529185360896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _step‚521 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚439436116017558 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99180682 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚6786.454980894 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140950003 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140926961 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚493353949440000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚493353925499904 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63794563 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99180109 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252188985 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚444305874091520 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚444305856088576 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111262026 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6349 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6529185360896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111262026 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚432929665425792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚450835832110592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚500160921066496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252212029 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚571 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252188985 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252212029 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚444305856088576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140950003 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140926961 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6807198084096 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63794565 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35385546 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439435328329578 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111262024 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252212029 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚444305856088576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140950003 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140926961 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6807198084096 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63794565 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35385546 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439435328329578 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111262024 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6529185360896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _step‚521 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚439436116017558 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99180682 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚6786.454980894 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140950003 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140926961 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚493353949440000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚493353925499904 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63794563 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99180109 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252188985 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚444305874091520 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚444305856088576 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111262026 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6349 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6529185360896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111262026 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚432929665425792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚450835832110592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚500160921066496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252212029 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚571 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252188985 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142017168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6578432315392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚503904287775232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254106020 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚503904505857536 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454201585431040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447623153115648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447623171118592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚497045994296320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚442721653127048 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454202376078848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447623171118592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚524 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454201585431040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6555520036788 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64267317 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6858487621120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112088850 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35646740 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6858293478912 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447623153115648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35646192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6579204960256 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112088852 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005“Š(ÿ'©ùO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚436166920778032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447623171118592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚524 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454201585431040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6555520036788 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64267317 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6858487621120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112088850 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35646740 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6858293478912 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447623153115648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35646192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6579204960256 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112088852 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚442722440814820 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112088852 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141994152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99913507 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142017168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6578432315392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚503904287775232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254106020 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚503904505857536 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454201585431040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447623153115648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447623171118592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚497045994296320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚442721653127048 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454202376078848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254144631 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35651980 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚442788720702686 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99928945 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447690853054976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6580209719296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚503980896107008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚497121342639104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99928388 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254121587 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35651425 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚497121366579200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6556521176874 _step‚525 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447690871057920 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112105724 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚6399 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚454271080777216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6859529527808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚555 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6579437061120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚436232970174432 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112105726 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254121587 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0¾ðÓ#ø¬‚N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254144631 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35651980 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚442788720702686 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99928945 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447690853054976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6580209719296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚503980896107008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚497121342639104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99928388 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254121587 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35651425 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚497121366579200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6556521176874 _step‚525 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447690871057920 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112105724 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚6399 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚454271080777216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6859529527808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚555 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6579437061120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚436232970174432 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112105726 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254121587 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142038905 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚557 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447690871057920 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457636834097664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143106070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112932552 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143106070 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚507724335709696 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143083054 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100662356 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚446075045500156 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚500813435375616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451008168084992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451008150082048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6910900334080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚439470225526672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 _step‚528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100661809 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451008168084992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112932552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35912639 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 _runtime‚ 6886.92831201 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256015604 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112932550 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457637624745472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64749717 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457636834097664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚507724117627392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚6449>•ç•&½úO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 _runtime‚ 6886.92831201 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256015604 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112932550 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457637624745472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64749717 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457636834097664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚507724117627392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚6449 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143083054 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256038622 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35912094 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884947872875113e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457637624745472 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6629456660480 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6628684015616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500813411435520 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6910706191872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6628684015616 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112932550 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451008150082048 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6604819973484 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457636834097664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143106070 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112932552 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143106070 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚507724335709696 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143083054 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100662356 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚446075045500156 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚500813435375616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451008168084992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451008150082048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6910900334080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚439470225526672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 _step‚528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100661809 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451008168084992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112932552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35912639 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100676696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚507800505651712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚446142900763774 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64759363 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚446142113075794 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 % _timestamp‚1.7884947883350089e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451075868024320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143104765 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256077233 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35917333 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6911745873408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6629688761344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500888783718400 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚451075850021376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6449 _runtime‚ 6887.97643815 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚555 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112949426 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256077233 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112949424 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64759365 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚439536309001792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6629688761344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451075868024320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚439536291962432 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100677251 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992RN¸9 'À„N [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100676696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚507800505651712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚446142900763774 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64759363 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚446142113075794 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 % _timestamp‚1.7884947883350089e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451075868024320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143104765 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256077233 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35917333 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6911745873408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6629688761344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500888783718400 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚451075850021376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6449 _runtime‚ 6887.97643815 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚555 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112949426 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256077233 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112949424 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64759365 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚439536309001792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6629688761344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451075868024320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚439536291962432 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100677251 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992‘¯#[¿2Á- _wandbruntime‚6887  _runtime‚6887Qÿ.wYÂÓ /loss/FRt1/assignment_loss‚2.434788830578327 0loss/FRt2/assignment_loss‚3.6658702257907754 /loss/FRt4/assignment_loss‚1.135969329264856 0loss/SRqqt2/detection_loss‚2.258483431031627 1loss/SRqqt3/detection_loss‚1.6735664646615063 1loss/SRqqt4/detection_loss‚0.8634510264290791 /loss/FRt2/detection_loss‚2.2677690317976187 $loss/total_loss‚40.4540795135086 % _timestamp‚1.7884947883374636e+09 2loss/SRqqt2/assignment_loss‚1.6304349714229185 .loss/FBt3/detection_loss‚2.324673056602478 /loss/FBt4/detection_loss‚0.3807969729726513 _step‚530 1loss/FBt1/assignment_loss‚0.12577129937679388 .loss/FRt3/detection_loss‚2.609880837447503 /loss/FBt1/detection_loss‚2.7421477526426314 trainer/global_step‚6449 _runtime‚6887.977698345 0loss/FBt2/assignment_loss‚0.2677491188690813 0loss/FBt3/assignment_loss‚0.1752690287429197 2loss/SRqqt1/assignment_loss‚1.4174260219389743 2loss/SRqqt3/assignment_loss‚1.6236249545687123 2loss/SRqqt4/assignment_loss‚0.7313320555258542 1loss/FBt4/assignment_loss‚0.15446522035086346 .loss/FRt1/detection_loss‚2.459258769173175 .loss/FRt4/detection_loss‚1.461034114742952 1loss/SRqqt1/detection_loss‚2.7509062547575343 /loss/FBt2/detection_loss‚2.5987190869119434 0loss/FRt3/assignment_loss‚2.7006916579078224 epoch‚5?¿žß^ÃØ 1loss/SRqqt3/detection_loss‚1.6735664646615063 1loss/SRqqt4/detection_loss‚0.8634510264290791 /loss/FRt2/detection_loss‚2.2677690317976187 $loss/total_loss‚40.4540795135086 % _timestamp‚1.7884947883374636e+09 2loss/SRqqt2/assignment_loss‚1.6304349714229185 .loss/FBt3/detection_loss‚2.324673056602478 /loss/FBt4/detection_loss‚0.3807969729726513 _step‚530 1loss/FBt1/assignment_loss‚0.12577129937679388 .loss/FRt3/detection_loss‚2.609880837447503 /loss/FBt1/detection_loss‚2.7421477526426314 trainer/global_step‚6449 _runtime‚6887.977698345 0loss/FBt2/assignment_loss‚0.2677491188690813 0loss/FBt3/assignment_loss‚0.1752690287429197 2loss/SRqqt1/assignment_loss‚1.4174260219389743 2loss/SRqqt3/assignment_loss‚1.6236249545687123 2loss/SRqqt4/assignment_loss‚0.7313320555258542 1loss/FBt4/assignment_loss‚0.15446522035086346 .loss/FRt1/detection_loss‚2.459258769173175 .loss/FRt4/detection_loss‚1.461034114742952 1loss/SRqqt1/detection_loss‚2.7509062547575343 /loss/FBt2/detection_loss‚2.5987190869119434 0loss/FRt3/assignment_loss‚2.7006916579078224 epoch‚5 /loss/FRt1/assignment_loss‚2.434788830578327 0loss/FRt2/assignment_loss‚3.6658702257907754 /loss/FRt4/assignment_loss‚1.135969329264856 0loss/SRqqt2/detection_loss‚2.258483431031627’äeSXÄ‚(:’ ЇéÔ؇௠gpu.0.gpu‚29.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.776999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ìnzÅ‚(:ï ЇéÔ””ÿ°# disk.dm-0.out‚168503.9755859375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.out‚0) disk./.usagePercent‚78.84161577783036% disk./.usageGB‚1174.7970504760742& disk.nvme0n1.out‚168503.9755859375 disk.drbd1260.out‚0.984375 disk.sda.out‚ 521.46875 disk.sda1.in‚0 network.sent‚ 1.5880411e+07 disk.sda2.out‚0( disk.nvme0c0n1.out‚168503.9755859375 disk.sda3.out‚ 521.46875 disk.dm-1.in‚0( proc.memory.availableMB‚ 1119.4921875 proc.cpu.threads‚17 disk.sda2.in‚0$ disk.nvme0c0n1.in‚24677.47265625" disk.nvme0n1.in‚24677.47265625 disk.dm-1.out‚ 521.46875$ memory_percent‚93.16716194152832! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 cpu‚25.281326536340266 network.recv‚ 2.26926e+06 disk.dm-0.in‚24677.47265625)hw1‘Æ‚(:† ߇éÔ»‰³ gpu.0.gpu‚7.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.836' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.1672 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z¬Ó<ƒÇ‚(:ø ߇éÔåÇ·) disk./.usagePercent‚78.84161577783036 disk.sda.out‚ 521.859375 disk.sda2.out‚0' disk.nvme0n1.out‚168880.97412109375$ disk.dm-0.out‚168880.97412109375) proc.memory.availableMB‚ 1119.48828125% disk./.usageGB‚1174.7970504760742 disk.sda2.in‚0$ memory_percent‚93.16718578338623* proc.memory.percent‚18.003177642822266 cpu‚25.28099585311263 proc.cpu.threads‚17 network.sent‚ 1.5988651e+07 disk.sda3.out‚ 521.859375 disk.dm-1.in‚0 disk.dm-1.out‚ 521.859375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.98828125$ disk.nvme0c0n1.in‚24677.47265625 disk.dm-0.in‚24677.47265625 network.recv‚ 2.275403e+06 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚168880.97412109375" disk.nvme0n1.in‚24677.47265625! proc.memory.rssMB‚ 2949.640625 disk.sda1.out‚0…X‘’È‚(:‡ î‡éÔ±¾”¯ gpu.0.gpu‚34.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.754' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1508 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TfÅÝtÉ‚(:é î‡éÔ„ŒÛ²% disk./.usageGB‚1174.7970504760742 disk.sda.in‚0 disk.sda.out‚522.3125 disk.nvme0n1.in‚ 24833.96875$ memory_percent‚93.16868782043457( proc.memory.availableMB‚ 1119.2421875) disk./.usagePercent‚78.84161577783036 disk.sda3.in‚0 disk.sda3.out‚522.3125 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.98828125 disk.sda1.out‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚169210.67431640625 network.sent‚ 1.5993694e+07 disk.dm-0.in‚ 24833.96875$ disk.dm-0.out‚169210.67431640625 disk.dm-1.out‚522.3125 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 24833.96875 network.recv‚ 2.278767e+06' disk.nvme0n1.out‚169210.67431640625! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 cpu‚25.280680362565985 proc.cpu.threads‚17¼n·í2Ê- _wandbruntime‚6936  _runtime‚6936€Þ˜ÉËà ' lr-AdamW/pg2‚0.00041332819884819054 trainer/global_step‚6499 % _timestamp‚1.7884948373302097e+09 _runtime‚6936.970562919 _step‚531 ' lr-AdamW/pg1‚0.00041332819884819054"ÖçDÎÌÈ ' lr-AdamW/pg1‚0.00041332819884819054 ' lr-AdamW/pg2‚0.00041332819884819054 trainer/global_step‚6499 % _timestamp‚1.7884948373302097e+09 _runtime‚6936.970562919 _step‚531“G¯Ý÷2Í- _wandbruntime‚6936  _runtime‚6936fo—ðJÎöO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113776252 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144171956 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚461072082764288 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113776252 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113776250 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 % _timestamp‚1.7884948373310049e+09 _runtime‚6936.971509192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚454393165051392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144194972 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101411464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚449429225561036 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36179347 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6654890558592 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6678935715840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚454393147048448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6679708360704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442773547314672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6963543521792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6678935715840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257971224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6963345185280 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚461072082764288 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113776252 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113776250 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 % _timestamp‚1.7884948373310049e+09 _runtime‚6936.971509192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚454393165051392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144194972 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101411464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚449429225561036 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36179347 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6654890558592 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6678935715840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚454393147048448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6679708360704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442773547314672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6963543521792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6678935715840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257971224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6963345185280 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101425899 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚454460864990720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6655891698470 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257986791 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚461141578110464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504656200857600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚504656176917504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 _runtime‚6937.822978072 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚454460864990720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257986791 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884948381817408e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚511620568137216 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258009835 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚442839630789792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6964391219712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6679940461568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65241765 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449496293136882 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6656662347090 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6679940461568 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101426444 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65241763 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6964585348608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258009835 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511620786206208 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449495505448902 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113793124 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144193667 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚461140787449344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113793126 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚6499 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258009835 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚442839630789792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6964391219712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6679940461568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65241765 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449496293136882 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6656662347090 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6679940461568 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101426444 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65241763 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6964585348608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258009835 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511620786206208 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449495505448902 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113793124 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144193667 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚461140787449344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113793126 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚6499 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101425899 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚454460864990720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6655891698470 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257986791 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚461141578110464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504656200857600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚504656176917504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 _runtime‚6937.822978072 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚454460864990720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257986791 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846•û×I2Ó- _wandbruntime‚6937  _runtime‚6937›ï XÔÒ /loss/FRt3/assignment_loss‚2.247328147623274 0loss/FRt4/assignment_loss‚1.0232760980725288 1loss/FBt2/assignment_loss‚0.08681888657702075 .loss/FRt3/detection_loss‚2.370780786044068 .loss/FBt1/detection_loss‚2.286165101961656 1loss/FBt4/assignment_loss‚0.10086425254685613 .loss/FRt2/detection_loss‚2.844792416957495 _runtime‚6937.824711629 2loss/SRqqt4/assignment_loss‚0.8685025741473179 /loss/FRt4/detection_loss‚1.4372876502540228 .loss/FBt2/detection_loss‚2.690205319361253 epoch‚5 % _timestamp‚1.7884948381844354e+09 /loss/FRt2/assignment_loss‚3.569779868479128 .loss/FRt1/detection_loss‚2.459180921316147 %loss/total_loss‚40.62864752764652 /loss/FBt1/assignment_loss‚0.342841752381322 1loss/FBt3/assignment_loss‚0.19415874659504576 0loss/SRqqt2/detection_loss‚2.715709294591631 trainer/global_step‚6499 _step‚534 /loss/FRt1/assignment_loss‚3.170125487446785 1loss/SRqqt1/detection_loss‚2.8235407417470757 1loss/SRqqt4/detection_loss‚0.7068553735417399 0loss/FBt4/detection_loss‚0.46258061186100047 2loss/SRqqt1/assignment_loss‚1.0787857081741095 1loss/SRqqt2/assignment_loss‚1.789130553070988 2loss/SRqqt3/assignment_loss‚1.1200997174243654 1loss/SRqqt3/detection_loss‚1.3684475621894787 /loss/FBt3/detection_loss‚2.8713899552822113¿‘q]Õ× /loss/FRt3/assignment_loss‚2.247328147623274 0loss/FRt4/assignment_loss‚1.0232760980725288 1loss/FBt2/assignment_loss‚0.08681888657702075 .loss/FRt3/detection_loss‚2.370780786044068 .loss/FBt1/detection_loss‚2.286165101961656 1loss/FBt4/assignment_loss‚0.10086425254685613 .loss/FRt2/detection_loss‚2.844792416957495 _runtime‚6937.824711629 2loss/SRqqt4/assignment_loss‚0.8685025741473179 /loss/FRt4/detection_loss‚1.4372876502540228 .loss/FBt2/detection_loss‚2.690205319361253 epoch‚5 % _timestamp‚1.7884948381844354e+09 /loss/FRt2/assignment_loss‚3.569779868479128 .loss/FRt1/detection_loss‚2.459180921316147 %loss/total_loss‚40.62864752764652 /loss/FBt1/assignment_loss‚0.342841752381322 1loss/FBt3/assignment_loss‚0.19415874659504576 0loss/SRqqt2/detection_loss‚2.715709294591631 trainer/global_step‚6499 _step‚534 /loss/FRt1/assignment_loss‚3.170125487446785 1loss/SRqqt1/detection_loss‚2.8235407417470757 1loss/SRqqt4/detection_loss‚0.7068553735417399 0loss/FBt4/detection_loss‚0.46258061186100047 2loss/SRqqt1/assignment_loss‚1.0787857081741095 1loss/SRqqt2/assignment_loss‚1.789130553070988 2loss/SRqqt3/assignment_loss‚1.1200997174243654 1loss/SRqqt3/detection_loss‚1.3684475621894787 /loss/FBt3/detection_loss‚2.8713899552822113–«S+|’Ö‚(:‡ ý‡éÔô»¬³ gpu.0.gpu‚40.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.004' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A·L}ׂ(:ò ý‡éÔ’‚²µ network.sent‚ 1.5999892e+07 network.recv‚ 2.28341e+06) disk.nvme0c0n1.out‚169474.75146484375 disk.nvme0n1.in‚ 24833.96875' disk.nvme0n1.out‚169474.75146484375( proc.memory.availableMB‚ 1119.4296875 cpu‚25.280274641813456 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 522.75390625 disk.dm-1.out‚ 522.75390625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0( disk./.usagePercent‚78.8417468536879% disk./.usageGB‚1174.7990036010742 disk.sda.out‚ 522.75390625! disk.nvme0c0n1.in‚ 24833.96875 disk.dm-0.in‚ 24833.96875$ disk.dm-0.out‚169474.75146484375$ memory_percent‚93.16754341125488! proc.memory.rssMB‚ 2949.640625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.9921875 disk.sda1.in‚0* proc.memory.percent‚18.003177642822266™¾7÷Ø‚(:’ ŒˆéÔ¡­Ü® gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚133.087' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.617399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëˆc|Ù‚(:ñ ŒˆéÔò¢Ç¯ cpu‚25.279933926388605) disk./.usagePercent‚78.84175402189885% disk./.usageGB‚1174.7991104125977 disk.nvme0n1.in‚ 24833.96875' disk.nvme0n1.out‚169899.22021484375 disk.dm-0.in‚ 24833.96875$ disk.dm-0.out‚169899.22021484375 disk.dm-1.out‚ 523.47265625 disk.drbd1260.in‚0 disk.sda3.out‚ 523.47265625 disk.drbd1260.out‚ 0.9921875 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.6109025e+07 network.recv‚ 2.290015e+06 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 24833.96875) disk.nvme0c0n1.out‚169899.22021484375 disk.sda3.in‚0 disk.sda.out‚ 523.47265625 disk.sda2.out‚0$ memory_percent‚93.16749572753906% proc.memory.availableMB‚ 1119.4375! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266§ ¡Ú‚(:’ ›ˆéÔ„™¦® gpu.0.gpu‚48.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.322' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.464400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€½7ÓpÛ‚(:å ›ˆéÔ³§Ñ¯ disk.sda3.in‚0 disk.dm-1.out‚523.875 disk.drbd1260.out‚ 0.99609375$ memory_percent‚93.16906929016113 network.recv‚ 2.293602e+06 disk.sda.out‚523.875 disk.sda1.in‚0 disk.dm-0.in‚ 24833.96875$ disk.dm-0.out‚170300.05517578125 disk.drbd1260.in‚0! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 disk.sda.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1119.1796875 cpu‚25.27955083788429 proc.cpu.threads‚17 network.sent‚ 1.6116147e+07% disk./.usageGB‚1173.7633781433105! disk.nvme0c0n1.in‚ 24833.96875 disk.nvme0n1.in‚ 24833.96875' disk.nvme0n1.out‚170300.05517578125 disk.sda3.out‚523.875 disk.dm-1.in‚0) disk./.usagePercent‚78.77224516026973 disk.sda2.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚170300.05517578125×ây2Ü- _wandbruntime‚6984  _runtime‚69841'ÏSÉÝà _step‚535 ' lr-AdamW/pg1‚0.00040761354334590985 ' lr-AdamW/pg2‚0.00040761354334590985 trainer/global_step‚6549 % _timestamp‚1.7884948849943457e+09 _runtime‚6984.634788041Œc3AÎÞÈ _runtime‚6984.634788041 _step‚535 ' lr-AdamW/pg1‚0.00040761354334590985 ' lr-AdamW/pg2‚0.00040761354334590985 trainer/global_step‚6549 % _timestamp‚1.7884948849943457e+09—£n{32ß- _wandbruntime‚6984  _runtime‚6984϶ ©ü'àöO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚446076886142032 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚464507331430912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚457778144014848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6705731792112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36445331 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6704961143700 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183656448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114619952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145283874 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102160370 _step‚536 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚457778144014848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65714517 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114619950 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114619952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145260858 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 _runtime‚6984.636008561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚515364067385344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114619950 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚515364274981888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6729960060928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7015821671424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 259880808 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464507331430912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚464508122078720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 % _timestamp‚1.7884948849955187e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36445853 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65714515 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚508348269654016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 trainer/global_step‚6549 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚457778162017792 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6729960060928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7016005327872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚452782617934144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚452781830246372 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772Ì[C(áûO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚446076886142032 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚464507331430912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚457778144014848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6705731792112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36445331 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6704961143700 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183656448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114619952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145283874 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102160370 _step‚536 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚457778144014848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65714517 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114619950 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114619952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145260858 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 _runtime‚6984.636008561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚515364067385344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114619950 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚515364274981888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6729960060928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7015821671424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 259880808 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464507331430912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚464508122078720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 % _timestamp‚1.7884948849955187e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36445853 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65714515 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚508348269654016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 trainer/global_step‚6549 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚457778162017792 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6729960060928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7016005327872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚452782617934144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚452781830246372 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772˜7óæ-2â- _wandbruntime‚6985  _runtime‚6985o ¯×#ã€N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚508423617996800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚446142935538432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6706732932198 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259919393 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145282569 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102174811 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145305611 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114636826 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194128896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65724163 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚452849685509990 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145305611 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚464576826777088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 259942437 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚457845843954176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 trainer/global_step‚6549 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259919393 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 _runtime‚6985.506609737 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚452848897822010 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36450648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚446142952577792 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114636824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65724165 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚457845861957120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7016876466176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114636826 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6730192161792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚464576826777088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6730964819968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚515440688591872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884948858656747e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259942437 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚464576036115968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36451195P v 'ä…N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36450648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚446142952577792 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114636824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65724165 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚457845861957120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7016876466176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114636826 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6730192161792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚464576826777088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6730964819968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚515440688591872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884948858656747e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259942437 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚464576036115968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36451195 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚508423617996800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚446142935538432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6706732932198 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259919393 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145282569 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102174811 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145305611 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114636826 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194128896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65724163 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚452849685509990 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145305611 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚464576826777088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 259942437 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚457845843954176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 trainer/global_step‚6549 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259919393 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 _runtime‚6985.506609737 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚452848897822010 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0™Ì2Œ72å- _wandbruntime‚6985  _runtime‚6985å•9"\æÖ 0loss/FRt3/assignment_loss‚2.5435553440233556 1loss/SRqqt3/assignment_loss‚1.421281072574978 .loss/FRt1/detection_loss‚2.178899797718399 /loss/FRt4/detection_loss‚1.4619476885534823 0loss/SRqqt1/detection_loss‚2.251157739470082 0loss/FRt2/assignment_loss‚3.1675970613956452 /loss/FRt3/detection_loss‚2.2840919094841654 1loss/SRqqt4/detection_loss‚1.2027143881624114 .loss/FBt1/detection_loss‚2.323352803786596 2loss/SRqqt1/assignment_loss‚1.6493376433368652 /loss/FBt3/detection_loss‚2.6957406282424925 _runtime‚6985.508074407 0loss/FRt1/assignment_loss‚3.3845883424465475 1loss/SRqqt3/detection_loss‚1.5134484369307757 _step‚538 0loss/FRt4/assignment_loss‚1.1139833490053812 epoch‚5 2loss/SRqqt4/assignment_loss‚1.0241694990224366 0loss/FBt4/assignment_loss‚0.1512400583951538 % _timestamp‚1.7884948858679216e+09 1loss/FBt1/assignment_loss‚0.36476922283569974 1loss/FBt3/assignment_loss‚0.09258510506042512 .loss/FRt2/detection_loss‚2.371919344433229 1loss/SRqqt2/detection_loss‚2.8965425274588843 /loss/FBt2/detection_loss‚3.2325421571731567 %loss/total_loss‚41.84947532442232 trainer/global_step‚6549 2loss/SRqqt2/assignment_loss‚1.8579618534581228 0loss/FBt2/assignment_loss‚0.1457209578657057 /loss/FBt4/detection_loss‚0.5203283935883327dWUaçÛ 1loss/FBt1/assignment_loss‚0.36476922283569974 1loss/FBt3/assignment_loss‚0.09258510506042512 .loss/FRt2/detection_loss‚2.371919344433229 1loss/SRqqt2/detection_loss‚2.8965425274588843 /loss/FBt2/detection_loss‚3.2325421571731567 %loss/total_loss‚41.84947532442232 trainer/global_step‚6549 2loss/SRqqt2/assignment_loss‚1.8579618534581228 0loss/FBt2/assignment_loss‚0.1457209578657057 /loss/FBt4/detection_loss‚0.5203283935883327 0loss/FRt3/assignment_loss‚2.5435553440233556 1loss/SRqqt3/assignment_loss‚1.421281072574978 .loss/FRt1/detection_loss‚2.178899797718399 /loss/FRt4/detection_loss‚1.4619476885534823 0loss/SRqqt1/detection_loss‚2.251157739470082 0loss/FRt2/assignment_loss‚3.1675970613956452 /loss/FRt3/detection_loss‚2.2840919094841654 1loss/SRqqt4/detection_loss‚1.2027143881624114 .loss/FBt1/detection_loss‚2.323352803786596 2loss/SRqqt1/assignment_loss‚1.6493376433368652 /loss/FBt3/detection_loss‚2.6957406282424925 _runtime‚6985.508074407 0loss/FRt1/assignment_loss‚3.3845883424465475 1loss/SRqqt3/detection_loss‚1.5134484369307757 _step‚538 0loss/FRt4/assignment_loss‚1.1139833490053812 epoch‚5 2loss/SRqqt4/assignment_loss‚1.0241694990224366 0loss/FBt4/assignment_loss‚0.1512400583951538 % _timestamp‚1.7884948858679216e+09š`¯¶’è‚(:‡ ªˆéÔØ«ü® gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.504' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N³´‰é‚(:þ ªˆéԧ߇± proc.cpu.threads‚17% disk./.usageGB‚1173.7392501831055 disk.sda3.out‚ 524.41015625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0# memory_percent‚93.1692361831665) proc.memory.availableMB‚ 1119.15234375 disk.dm-0.in‚24989.31640625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 524.41015625 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.in‚24989.31640625' disk.nvme0n1.out‚170622.57470703125 network.recv‚ 2.296931e+06) disk./.usagePercent‚78.77062591261566$ disk.dm-0.out‚170622.57470703125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.99609375$ disk.nvme0c0n1.in‚24989.31640625* proc.memory.percent‚18.003177642822266 cpu‚25.279123821418654 network.sent‚ 1.6132345e+07 disk.dm-1.out‚ 524.41015625) disk.nvme0c0n1.out‚170622.57470703125! proc.memory.rssMB‚ 2949.640625ËÏý’ê‚(:‡ ¹ˆéÔçË¢¯ gpu.0.gpu‚26.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îô‰†ë‚(:û ¹ˆéÔ›ïѲ% disk./.usageGB‚1173.7392501831055 disk.sda3.out‚ 524.9453125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.99609375 disk.sda1.in‚0" disk.nvme0n1.in‚24989.31640625 disk.sda2.in‚0$ disk.dm-0.out‚170903.24365234375! proc.memory.rssMB‚ 2949.640625 cpu‚25.278662453423205 network.sent‚ 1.6228687e+07 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 524.9453125 disk.sda1.out‚0' disk.nvme0n1.out‚170903.24365234375# memory_percent‚93.1678056716919) proc.memory.availableMB‚ 1119.38671875 disk.dm-1.out‚ 524.9453125 disk.sda.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚24989.31640625) disk.nvme0c0n1.out‚170903.24365234375 disk.dm-0.in‚24989.31640625* proc.memory.percent‚18.003177642822266 proc.cpu.threads‚17 network.recv‚ 2.303617e+06) disk./.usagePercent‚78.77062591261566ËAœ’ì‚(:‡ ȈéÔ¼ÀÙ° gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚130.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0604 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö\Àuí‚(:ê ȈéÔ˜÷´ disk.drbd1260.in‚0 disk.drbd1260.out‚1 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 525.40625 proc.cpu.threads‚17 network.sent‚ 1.6235605e+07 network.recv‚ 2.307265e+06% disk./.usageGB‚1173.7392501831055 disk.dm-1.in‚0 disk.sda.out‚ 525.40625 disk.sda1.out‚0) disk.nvme0c0n1.out‚171230.55224609375 disk.dm-0.in‚24989.31640625 disk.dm-1.out‚ 525.40625 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚24989.31640625" disk.nvme0n1.in‚24989.31640625$ disk.dm-0.out‚171230.55224609375( proc.memory.availableMB‚ 1119.3828125) disk./.usagePercent‚78.77062591261566' disk.nvme0n1.out‚171230.55224609375# memory_percent‚93.1678295135498! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 cpu‚25.27827404608073–ª¸2î- _wandbruntime‚7030  _runtime‚7030F‘ØÈï ' lr-AdamW/pg1‚0.00040189649282610117 ' lr-AdamW/pg2‚0.00040189649282610117 trainer/global_step‚6599 % _timestamp‚1.7884949310241468e+09 _runtime‚ 7030.66442652 _step‚539ôÎœ ÍðÇ ' lr-AdamW/pg1‚0.00040189649282610117 ' lr-AdamW/pg2‚0.00040189649282610117 trainer/global_step‚6599 % _timestamp‚1.7884949310241468e+09 _runtime‚ 7030.66442652 _step‚539›ÚM›2ñ- _wandbruntime‚7030  _runtime‚7030u"€òöO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449380190890672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6755802377220 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚467942580097536 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146349760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66196915 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚461163158984192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115463652 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102909271 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6780211761152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146372776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚467942580097536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 _runtime‚7030.665529098 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 261836428 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36711802 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66196917 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚512115662853120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7068541316096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚461163140981248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚467943370745344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6779439116288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115463650 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚6599 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115463652 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449380207930032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6779439116288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461163158984192›°(óûO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146372776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚467942580097536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 _runtime‚7030.665529098 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 261836428 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36711802 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66196917 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚512115662853120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7068541316096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚461163140981248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚467943370745344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6779439116288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115463650 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚6599 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115463652 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449380207930032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6779439116288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461163158984192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449380190890672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6755802377220 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚467942580097536 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146349760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66196915 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚461163158984192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115463652 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102909271 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6780211761152œBH†V2ô- _wandbruntime‚7031  _runtime‚7031šMë'õÿM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468012075443712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚512191011195904 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36717658 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 261851995 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7069409882112 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115480524 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚512191035136000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115480526 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449446274365792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚554 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102924223 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6781216520192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456203077883098 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66206565 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146371471 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _runtime‚7031.522414236 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 261851995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146394513 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 261875039 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115480524 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788494931881506e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚468012075443712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚461230858923520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6756032868686 _step‚541 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102923669 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚461230840920576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚461230840920576 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115480526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66206563 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚519260634952704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005~ùL[)&ö„N TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66206565 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146371471 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _runtime‚7031.522414236 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 261851995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146394513 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 261875039 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115480524 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788494931881506e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚468012075443712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚461230858923520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6756032868686 _step‚541 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102923669 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚461230840920576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚461230840920576 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115480526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66206563 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚519260634952704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468012075443712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚512191011195904 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36717658 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 261851995 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7069409882112 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115480524 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚512191035136000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115480526 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449446274365792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚554 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102924223 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6781216520192 ICDeviceStatsMonitor.on_traiŠiG/án_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4562030778830986?2÷- _wandbruntime‚7031  _runtime‚7031qFÕ¯UøÏ .loss/FRt2/assignment_loss‚4.09201789855957 /loss/FRt4/assignment_loss‚1.725791280684264 .loss/FBt1/detection_loss‚2.339491218328476 /loss/FRt1/assignment_loss‚3.516002983517117 0loss/FBt1/assignment_loss‚0.3811198901966719 1loss/FBt3/assignment_loss‚0.49787632022853356 1loss/SRqqt1/assignment_loss‚1.082799632398581 0loss/SRqqt2/detection_loss‚2.687608277356183 %loss/total_loss‚42.04032453419849 % _timestamp‚1.7884949318839896e+09 /loss/FRt4/detection_loss‚1.7071237044975809 0loss/SRqqt3/detection_loss‚1.325714837597764 0loss/SRqqt4/detection_loss‚1.055022937740432 /loss/FBt4/detection_loss‚0.4889667044187998 2loss/SRqqt3/assignment_loss‚1.1782450922157452 /loss/FRt2/detection_loss‚2.8886082162635245 _runtime‚7031.524196745 _step‚542 0loss/FRt3/assignment_loss‚2.3055723171967726 1loss/SRqqt2/assignment_loss‚2.060177900173046 0loss/FBt2/assignment_loss‚0.3569943882786132 1loss/FBt4/assignment_loss‚0.10872934235499691 .loss/FRt1/detection_loss‚2.134041282865736 /loss/FRt3/detection_loss‚2.4807396810024214 epoch‚5 2loss/SRqqt4/assignment_loss‚1.0564241479733028 0loss/SRqqt1/detection_loss‚2.632993069562045 /loss/FBt2/detection_loss‚1.8547074382121747 /loss/FBt3/detection_loss‚2.0835559725761414 trainer/global_step‚6599²zËÊZùÔ /loss/FBt2/detection_loss‚1.8547074382121747 /loss/FBt3/detection_loss‚2.0835559725761414 trainer/global_step‚6599 .loss/FRt2/assignment_loss‚4.09201789855957 /loss/FRt4/assignment_loss‚1.725791280684264 .loss/FBt1/detection_loss‚2.339491218328476 /loss/FRt1/assignment_loss‚3.516002983517117 0loss/FBt1/assignment_loss‚0.3811198901966719 1loss/FBt3/assignment_loss‚0.49787632022853356 1loss/SRqqt1/assignment_loss‚1.082799632398581 0loss/SRqqt2/detection_loss‚2.687608277356183 %loss/total_loss‚42.04032453419849 % _timestamp‚1.7884949318839896e+09 /loss/FRt4/detection_loss‚1.7071237044975809 0loss/SRqqt3/detection_loss‚1.325714837597764 0loss/SRqqt4/detection_loss‚1.055022937740432 /loss/FBt4/detection_loss‚0.4889667044187998 2loss/SRqqt3/assignment_loss‚1.1782450922157452 /loss/FRt2/detection_loss‚2.8886082162635245 _runtime‚7031.524196745 _step‚542 0loss/FRt3/assignment_loss‚2.3055723171967726 1loss/SRqqt2/assignment_loss‚2.060177900173046 0loss/FBt2/assignment_loss‚0.3569943882786132 1loss/FBt4/assignment_loss‚0.10872934235499691 .loss/FRt1/detection_loss‚2.134041282865736 /loss/FRt3/detection_loss‚2.4807396810024214 epoch‚5 2loss/SRqqt4/assignment_loss‚1.0564241479733028 0loss/SRqqt1/detection_loss‚2.632993069562045žÅúê’ú‚(:‡ ׈éÔ򪓡 gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.641' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1282 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆö ¦€û‚(:õ ׈éÔû뉳) disk.nvme0c0n1.out‚171518.35791015625" disk.nvme0n1.in‚24989.31640625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚93.16818714141846% disk./.usageGB‚1173.7627639770508 disk.drbd1260.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 network.sent‚ 1.6238683e+07 network.recv‚ 2.310553e+06' disk.nvme0n1.out‚171518.35791015625 disk.dm-0.in‚24989.31640625 disk.drbd1260.out‚1 disk.sda.out‚ 525.80859375 disk.sda2.out‚0) proc.memory.availableMB‚ 1119.32421875) disk./.usagePercent‚78.77220394305672$ disk.nvme0c0n1.in‚24989.31640625$ disk.dm-0.out‚171518.35791015625 disk.sda3.out‚ 525.80859375 disk.dm-1.out‚ 525.80859375 cpu‚25.27783576152485 proc.cpu.threads‚17d6’ü‚(:‡ æˆéÔ·¼Û® gpu.0.gpu‚38.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.964' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Kpý‚(:å æˆéÔ„œÁ² disk.nvme0n1.in‚25111.5 disk.dm-0.in‚25111.5 proc.cpu.threads‚17 network.sent‚ 1.6346711e+07 network.recv‚ 2.316483e+06 disk.sda3.in‚0 disk.sda3.out‚ 526.3671875 disk.sda2.out‚0' disk.nvme0n1.out‚171869.74072265625$ disk.dm-0.out‚171869.74072265625$ memory_percent‚93.16821098327637) disk./.usagePercent‚78.77220291902658 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1119.3203125! proc.memory.rssMB‚ 2949.640625 cpu‚25.27746571448977 disk.sda1.in‚0% disk./.usageGB‚1173.7627487182617 disk.dm-1.out‚ 526.3671875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.00390625 disk.sda2.in‚0 disk.nvme0c0n1.in‚25111.5* proc.memory.percent‚18.003177642822266 disk.sda.out‚ 526.3671875) disk.nvme0c0n1.out‚171869.74072265625·Mû þ‚(:’ õˆéÔ¬‡ž³ gpu.0.gpu‚37.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.478' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.695600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wóPHgÿ‚(:Ü õˆéÔû¸Ô» network.recv‚ 2.321367e+06$ disk./.usageGB‚1173.762710571289 disk.sda2.out‚0 disk.nvme0c0n1.in‚25111.5 disk.nvme0n1.in‚25111.5 disk.dm-0.in‚25111.5 disk.sda3.in‚0) disk./.usagePercent‚78.77220035895124) disk.nvme0c0n1.out‚172028.88916015625$ disk.dm-0.out‚172028.88916015625 disk.sda3.out‚ 526.90625 disk.dm-1.in‚0 disk.dm-1.out‚ 526.90625 disk.drbd1260.out‚ 1.00390625$ memory_percent‚93.16825866699219' disk.nvme0n1.out‚172028.88916015625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 1119.3125 cpu‚25.277062078339803 disk.sda2.in‚0 disk.sda.out‚ 526.90625 disk.sda1.in‚0! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 proc.cpu.threads‚17 network.sent‚ 1.6352949e+07gMïØ2€- _wandbruntime‚7079  _runtime‚7079®ÊÈ _step‚543 ' lr-AdamW/pg1‚0.00039617823871352373 ' lr-AdamW/pg2‚0.00039617823871352373 trainer/global_step‚6649 $ _timestamp‚1.788494979599684e+09 _runtime‚7079.240079837êÒÔµÍ‚Ç ' lr-AdamW/pg1‚0.00039617823871352373 ' lr-AdamW/pg2‚0.00039617823871352373 trainer/global_step‚6649 $ _timestamp‚1.788494979599684e+09 _runtime‚7079.240079837 _step‚543Ÿ:v’2ƒ- _wandbruntime‚7079  _runtime‚7079¿ú™nû'„õO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36979816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523003974797312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103658581 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452683512678672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263746012 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147461678 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103659133 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚6649 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459488614992588 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚515883079992320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36979266 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _runtime‚7079.241373364 _step‚544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6830463461376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788494979600894e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66679315 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147438662 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6829690816512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66679317 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523004197073920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚515883103932416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471377828764160 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459489402680360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464548155950592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6829690816512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116307350 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6805102313916 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 263769030 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116307352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚471378619411968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6830463461376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464548155950592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36979266 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _runtime‚7079.241373364 _step‚544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6830463461376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788494979600894e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66679315 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147438662 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6829690816512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66679317 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523004197073920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚515883103932416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471377828764160 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459489402680360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464548155950592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6829690816512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116307350 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6805102313916 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 263769030 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116307352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚471378619411968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6830463461376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464548155950592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452683512678672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263746012 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147461678 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103659133 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚6649 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459488614992588 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚515883079992320 ¬S·„’†‚(:‡ „‰éÔó°â± gpu.0.gpu‚49.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©õVmf‡‚(:Û „‰éÔñÚº³ disk.sda2.in‚0 disk.dm-0.in‚25111.5$ memory_percent‚93.16978454589844 cpu‚25.276785094322666 proc.cpu.threads‚17) disk./.usagePercent‚78.77220035895124 disk.drbd1260.out‚ 1.0078125 disk.sda1.out‚0 disk.nvme0c0n1.in‚25111.5& disk.nvme0n1.out‚172308.4443359375* proc.memory.percent‚18.003177642822266$ disk./.usageGB‚1173.762710571289 disk.sda3.in‚0 disk.sda3.out‚ 527.328125 disk.drbd1260.in‚0 disk.sda.out‚ 527.328125( disk.nvme0c0n1.out‚172308.4443359375% proc.memory.availableMB‚ 1119.0625 network.recv‚ 2.325886e+06 disk.dm-1.out‚ 527.328125 disk.sda.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚25111.5# disk.dm-0.out‚172308.4443359375! proc.memory.rssMB‚ 2949.640625 network.sent‚ 1.6358791e+07 disk.dm-1.in‚0 disk.sda1.in‚0Èæå32ˆ- _wandbruntime‚7080  _runtime‚7080Ñ•«| ‰€N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66688965 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263807641 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228554752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚515958428335104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6830695562240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6806874102414 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 % _timestamp‚1.7884949805373926e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263807641 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103673528 MGDeviceStatsMonitor.on_traS¹¸ÔŠin_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7122135316480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6830695562240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36984565 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 trainer/global_step‚6649 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147460373 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66688963 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263784597 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚471447324110336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116324224 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚523080359036928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459555682568226 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464615837886976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103674080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147483415 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚7080.178606831 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚523080587591680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚452749579114432 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚464615855889920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6831468220416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263784597 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 _step‚545 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116324226 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6831468220416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7121930701824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚515958452275200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147483415 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚464615837886976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116324224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚471446533449216 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚471447324110336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116324224 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚523080359036928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459555682568226 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464615837886976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103674080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147483415 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚7080.178606831 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚523080587591680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚452749579114432 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚464615855889920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6831468220416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263784597 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 _step‚545 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116324226 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6831468220416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7121930701824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚515958452275200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147483415 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚464615837886976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116324224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚471446533449216 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6830695562240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6806874102414 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 % _timestamp‚1.7884949805373926e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263807641 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103673528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7122135316480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6830695562240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36984565 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 trainer/global_step‚6649 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147460373 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66688963 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263784597 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680¡¼‘|y2‹- _wandbruntime‚7080  _runtime‚7080EIñ\ŒÖ 0loss/FRt4/assignment_loss‚1.6052948089896655 1loss/FBt1/assignment_loss‚0.32430330522340356 1loss/SRqqt4/detection_loss‚0.8230618106593427 /loss/FRt1/detection_loss‚2.1827379152751885 0loss/FRt3/assignment_loss‚1.8868636290232341 2loss/SRqqt2/assignment_loss‚1.7432790753664449 1loss/SRqqt3/detection_loss‚1.3249479387886822 2loss/SRqqt4/assignment_loss‚0.8675392102205047 -loss/FRt4/detection_loss‚1.49994902208936 .loss/FBt1/detection_loss‚2.301725374800818 epoch‚5 # _timestamp‚1.78849498054034e+09 _runtime‚7080.180591561 1loss/FBt4/assignment_loss‚0.18992134518354123 /loss/FRt3/detection_loss‚1.9198916812737783 /loss/FBt2/detection_loss‚2.6228552520275117 /loss/FBt3/detection_loss‚2.6074217915534974 1loss/FBt2/assignment_loss‚0.25396767566571726 %loss/total_loss‚39.21477093665743 0loss/FRt1/assignment_loss‚3.1243903098198085 0loss/FRt2/assignment_loss‚3.2091652477109753 1loss/FBt3/assignment_loss‚0.15564782172295963 trainer/global_step‚6649 .loss/FRt2/detection_loss‚2.480419664888707 0loss/SRqqt2/detection_loss‚2.457278400659561 /loss/FBt4/detection_loss‚0.6965828248853417 2loss/SRqqt1/assignment_loss‚1.6426609032787383 2loss/SRqqt3/assignment_loss‚1.0880933300440019 1loss/SRqqt1/detection_loss‚2.2067725975066423 _step‚5468XxxaÛ 0loss/FRt1/assignment_loss‚3.1243903098198085 0loss/FRt2/assignment_loss‚3.2091652477109753 1loss/FBt3/assignment_loss‚0.15564782172295963 trainer/global_step‚6649 .loss/FRt2/detection_loss‚2.480419664888707 0loss/SRqqt2/detection_loss‚2.457278400659561 /loss/FBt4/detection_loss‚0.6965828248853417 2loss/SRqqt1/assignment_loss‚1.6426609032787383 2loss/SRqqt3/assignment_loss‚1.0880933300440019 1loss/SRqqt1/detection_loss‚2.2067725975066423 _step‚546 0loss/FRt4/assignment_loss‚1.6052948089896655 1loss/FBt1/assignment_loss‚0.32430330522340356 1loss/SRqqt4/detection_loss‚0.8230618106593427 /loss/FRt1/detection_loss‚2.1827379152751885 0loss/FRt3/assignment_loss‚1.8868636290232341 2loss/SRqqt2/assignment_loss‚1.7432790753664449 1loss/SRqqt3/detection_loss‚1.3249479387886822 2loss/SRqqt4/assignment_loss‚0.8675392102205047 -loss/FRt4/detection_loss‚1.49994902208936 .loss/FBt1/detection_loss‚2.301725374800818 epoch‚5 # _timestamp‚1.78849498054034e+09 _runtime‚7080.180591561 1loss/FBt4/assignment_loss‚0.18992134518354123 /loss/FRt3/detection_loss‚1.9198916812737783 /loss/FBt2/detection_loss‚2.6228552520275117 /loss/FBt3/detection_loss‚2.6074217915534974 1loss/FBt2/assignment_loss‚0.25396767566571726 %loss/total_loss‚39.21477093665743¢ñrkp’Ž‚(:‡ “‰éÔ¡¾ã° gpu.0.gpu‚19.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A‹?6o‚(:ä “‰éÔá¸Í² disk.sda1.in‚0( disk.nvme0c0n1.out‚172735.1103515625* proc.memory.percent‚18.003177642822266 cpu‚25.276421570954653 network.sent‚ 1.6455646e+07) disk./.usagePercent‚78.76091503482685 disk.sda3.in‚0 disk.dm-1.out‚ 527.69140625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.0078125 disk.sda.in‚0 disk.sda2.out‚0# disk.dm-0.out‚172735.1103515625 disk.sda3.out‚ 527.69140625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚25111.5 proc.cpu.threads‚17 network.recv‚ 2.331986e+06% disk./.usageGB‚1173.5945510864258 disk.dm-0.in‚25111.5 disk.nvme0c0n1.in‚25111.5& disk.nvme0n1.out‚172735.1103515625# memory_percent‚93.1699275970459( proc.memory.availableMB‚ 1119.0390625! proc.memory.rssMB‚ 2949.640625 disk.sda.out‚ 527.69140625 8Óœ‚(:‘ ¢‰éÔ¦Ôÿ­ gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.72' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.544000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…¤Ræ‚‘‚(:÷ ¢‰éÔ䤂° network.recv‚ 2.337349e+06 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 25249.6796875( disk.nvme0c0n1.out‚172830.7197265625* proc.memory.percent‚18.003177642822266) disk./.usagePercent‚78.75934238054401 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 25249.6796875 disk.sda3.in‚0 disk.sda3.out‚ 528.41796875 cpu‚25.276002996077274$ disk./.usageGB‚1173.571117401123 disk.dm-1.out‚ 528.41796875 disk.sda.in‚0 disk.sda.out‚ 528.41796875& disk.nvme0n1.out‚172830.7197265625 disk.dm-0.in‚ 25249.6796875$ memory_percent‚93.17145347595215 proc.cpu.threads‚17 network.sent‚ 1.6553067e+07 disk.drbd1260.out‚ 1.01171875 disk.sda1.in‚0 disk.sda2.out‚0# disk.dm-0.out‚172830.7197265625( proc.memory.availableMB‚ 1118.7890625! proc.memory.rssMB‚ 2949.640625˜Ä±Ž’‚(:ƒ ±‰éÔž»ë¬ gpu.0.gpu‚47.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.3' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.86 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x†E§ƒ“‚(:ø ±‰éÔøß«® disk.drbd1260.out‚ 1.01171875 disk.sda2.out‚0( disk.nvme0c0n1.out‚173284.8095703125# disk.dm-0.out‚173284.8095703125) proc.memory.availableMB‚ 1119.28515625 network.sent‚ 1.6559206e+07% disk./.usageGB‚1173.7184143066406 disk.sda1.in‚0& disk.nvme0n1.out‚173284.8095703125! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 cpu‚25.27564872932244 proc.cpu.threads‚17 disk.dm-1.out‚ 528.98046875 disk.sda.in‚0 disk.sda.out‚ 528.98046875 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 25249.6796875$ memory_percent‚93.16842555999756 network.recv‚ 2.341952e+06) disk./.usagePercent‚78.76922759946365 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 25249.6796875 disk.dm-0.in‚ 25249.6796875 disk.sda3.in‚0 disk.sda3.out‚ 528.98046875 disk.dm-1.in‚0²7=2”- _wandbruntime‚7126  _runtime‚7126¾ÙœÃÆ•À & lr-AdamW/pg1‚0.0003904599726837641 & lr-AdamW/pg2‚0.0003904599726837641 trainer/global_step‚6699 $ _timestamp‚1.788495026755875e+09 _runtime‚7126.396210735 _step‚547¡k–Ë–Å & lr-AdamW/pg1‚0.0003904599726837641 & lr-AdamW/pg2‚0.0003904599726837641 trainer/global_step‚6699 $ _timestamp‚1.788495026755875e+09 _runtime‚7126.396210735 _step‚547£Æ@¤P2—- _wandbruntime‚7126  _runtime‚7126˜ŽÙç"˜öO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6880715161600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462842795053468 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚474813868078592 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265678614 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104407608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37246445 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519650497131520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884950267568858e+09 _step‚548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7173403948544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474813077430784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6879942516736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67161717 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6855172899024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚467933152916992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117151050 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117151052 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚7126.397432487 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265701632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6855943547436 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚474813077430784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67161715 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6880715161600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519650521071616 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467933152916992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6879942516736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467933134914048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_p|Ÿ‘4ool.current‚ 204627968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚462842007365696 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚526823901080064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455986834466672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚455986851506032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117151050 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚474813868078592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 1þó(™ûO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚467933134914048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117151052 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148527564 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104407608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37246445 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519650497131520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884950267568858e+09 _step‚548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7173403948544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474813077430784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6879942516736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67161717 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6855172899024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚467933152916992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117151050 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117151052 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚7126.397432487 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265701632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6855943547436 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚474813077430784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67161715 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6880715161600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519650521071616 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467933152916992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6879942516736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467933134914048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204627968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚462842007365696 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚526823901080064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455986834466672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚455986851506032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117151050 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚474813868078592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6880715161600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462842795053468 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚474813868078592 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265678614 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148549275 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117167926 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6881719920640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6856174038902 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚526900525239808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6856944687522 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67171363 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526900315559424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7174655825408 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚462909074941334 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265740243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519725845474304 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6880947262464 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚468000834853376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7174470085120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚474882572776960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚549 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚474881782115840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104422556 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚474882572776960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462909862629314 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519725869414400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825U¹$ 'œ…N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148572317 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148549275 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117167926 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6881719920640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6856174038902 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚526900525239808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6856944687522 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67171363 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526900315559424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7174655825408 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚462909074941334 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265740243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519725845474304 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6880947262464 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚468000834853376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7174470085120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚474882572776960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚549 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚474881782115840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104422556 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚474882572776960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462909862629314 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519725869414400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6880947262464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005¥o~â&2- _wandbruntime‚7127  _runtime‚7127íxé¡DžÚ /loss/FRt2/detection_loss‚2.7887867766045056 0loss/FBt2/assignment_loss‚0.3466030256758214 4loss/FBt3/assignment_loss‚2.1813693324061617e-05 0loss/FRt4/assignment_loss‚1.2405872612583393 /loss/FBt1/detection_loss‚2.9311297833919525 /loss/FBt2/detection_loss‚2.2705362168225376 0loss/FRt3/assignment_loss‚1.8079564977199474 2loss/SRqqt1/assignment_loss‚1.2860201997682452 2loss/SRqqt2/assignment_loss‚1.8057892360994894 _runtime‚7127.334984663 1loss/FBt1/assignment_loss‚0.29216908008529574 .loss/FRt1/detection_loss‚2.595706894993782 _step‚550 2loss/SRqqt3/assignment_loss‚1.3508827498682388 /loss/FRt3/detection_loss‚1.9421523048522624 1loss/SRqqt4/detection_loss‚1.0407840982079506 % _timestamp‚1.7884950276947763e+09 trainer/global_step‚6699 0loss/FBt4/assignment_loss‚0.1175630809574444 0loss/SRqqt3/detection_loss‚1.225238155391885 /loss/FBt3/detection_loss‚2.3779477422887627 /loss/FBt4/detection_loss‚0.6380045719561167 &loss/total_loss‚39.569063961588256 0loss/FRt2/assignment_lossÇCŒ‚3.3082909603272714 2loss/SRqqt4/assignment_loss‚0.9180136162322015 /loss/FRt4/detection_loss‚1.5323770433208164 /loss/SRqqt1/detection_loss‚2.36692354362458 epoch‚5 0loss/FRt1/assignment_loss‚3.1507942707915055 0loss/SRqqt2/detection_loss‚2.2347850376559840Â%eŸß 2loss/SRqqt3/assignment_loss‚1.3508827498682388 /loss/FRt3/detection_loss‚1.9421523048522624 1loss/SRqqt4/detection_loss‚1.0407840982079506 % _timestamp‚1.7884950276947763e+09 trainer/global_step‚6699 0loss/FBt4/assignment_loss‚0.1175630809574444 0loss/SRqqt3/detection_loss‚1.225238155391885 /loss/FBt3/detection_loss‚2.3779477422887627 /loss/FBt4/detection_loss‚0.6380045719561167 &loss/total_loss‚39.569063961588256 0loss/FRt2/assignment_loss‚3.3082909603272714 2loss/SRqqt4/assignment_loss‚0.9180136162322015 /loss/FRt4/detection_loss‚1.5323770433208164 /loss/SRqqt1/detection_loss‚2.36692354362458 epoch‚5 0loss/FRt1/assignment_loss‚3.1507942707915055 0loss/SRqqt2/detection_loss‚2.234785037655984 /loss/FRt2/detection_loss‚2.7887867766045056 0loss/FBt2/assignment_loss‚0.3466030256758214 4loss/FBt3/assignment_loss‚2.1813693324061617e-05 0loss/FRt4/assignment_loss‚1.2405872612583393 /loss/FBt1/detection_loss‚2.9311297833919525 /loss/FBt2/detection_loss‚2.2705362168225376 0loss/FRt3/assignment_loss‚1.8079564977199474 2loss/SRqqt1/assignment_loss‚1.2860201997682452 2loss/SRqqt2/assignment_loss‚1.8057892360994894 _runtime‚7127.334984663 1loss/FBt1/assignment_loss‚0.29216908008529574 .loss/FRt1/detection_loss‚2.595706894993782 _step‚550¦†ÔjÇ’ ‚(:‡ À‰éÔ¾õÙ° gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚117.827' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5654 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬m u~¡‚(:ó À‰éÔòÞ¤²$ memory_percent‚93.16873550415039' proc.memory.availableMB‚ 1119.234375! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 cpu‚25.2752550554122 disk.dm-1.out‚ 529.7890625 disk.sda.in‚0 disk.sda.out‚ 529.7890625 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 1.6568637e+07) disk./.usagePercent‚78.73101617091046 disk.sda3.in‚0 disk.sda3.out‚ 529.7890625 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 25249.6796875 network.recv‚ 2.353288e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚1.015625 disk.sda1.out‚0) disk.nvme0c0n1.out‚173846.64404296875 disk.dm-0.in‚ 25249.6796875$ disk.dm-0.out‚173846.64404296875% disk./.usageGB‚1173.1490364074707! disk.nvme0n1.in‚ 25249.6796875' disk.nvme0n1.out‚173846.64404296875¤¥j’¢‚(:‡ ωéÔ¤¾°¯ gpu.0.gpu‚16.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.507' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.5014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ·W‚£‚(:÷ ωéÔ‰ŽŒ° disk.dm-0.in‚ 25249.6796875$ disk.dm-0.out‚173941.99560546875' proc.memory.availableMB‚ 1118.796875 cpu‚25.27490782805851 disk.drbd1260.out‚1.015625 disk.sda.out‚ 530.18359375 network.recv‚ 2.359155e+06 disk.sda3.in‚0 disk.sda3.out‚ 530.18359375 disk.dm-1.out‚ 530.18359375 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 25249.6796875$ memory_percent‚93.17140579223633! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 proc.cpu.threads‚17 disk.sda.in‚0) disk./.usagePercent‚78.73090147953513% disk./.usageGB‚1173.1473274230957 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚173941.99560546875! disk.nvme0n1.in‚ 25249.6796875 network.sent‚ 1.6676826e+07 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0n1.out‚173941.99560546875§íUZ’¤‚(:‡ Þ‰éÔµÎô­ gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚125.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óد†¥‚(:û Þ‰éÔ³Ÿò® network.recv‚ 2.362528e+06% disk./.usageGB‚1173.1239013671875 disk.sda2.in‚0$ disk.nvme0c0n1.in‚25365.43359375) disk.nvme0c0n1.out‚174133.97119140625 disk.dm-0.in‚25365.43359375$ disk.dm-0.out‚174133.97119140625 disk.sda3.in‚0 network.sent‚ 1.6681839e+07 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0n1.out‚174133.97119140625 disk.dm-1.in‚0 disk.dm-1.out‚ 530.65234375 disk.sda.out‚ 530.65234375( proc.memory.availableMB‚ 1119.1796875 disk.sda2.out‚0" disk.nvme0n1.in‚25365.43359375 disk.sda3.out‚ 530.65234375 disk.sda.in‚0$ memory_percent‚93.16906929016113* proc.memory.percent‚18.003177642822266 cpu‚25.274623485479 proc.cpu.threads‚17) disk./.usagePercent‚78.72932933726736 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.01953125! proc.memory.rssMB‚ 2949.6406258)‰:’¦‚(:‡ í‰éÔÁ‡§­ gpu.0.gpu‚50.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.514' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÓe§‚(:ö í‰éÔÓ“•¯ disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚25365.43359375 network.sent‚ 1.6688106e+07 disk.drbd1260.out‚ 1.01953125 disk.sda.out‚ 531.09375) disk.nvme0c0n1.out‚174541.82275390625$ disk.dm-0.out‚174541.82275390625) proc.memory.availableMB‚ 1118.93359375! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266% disk./.usageGB‚1173.1553192138672 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚25365.43359375' disk.nvme0n1.out‚174541.82275390625$ memory_percent‚93.17057132720947 cpu‚25.274226589660266 disk.dm-1.out‚ 531.09375 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0n1.in‚25365.43359375 proc.cpu.threads‚17 network.recv‚ 2.367446e+06) disk./.usagePercent‚78.73143781531937 disk.sda3.in‚0 disk.sda3.out‚ 531.09375 disk.drbd1260.in‚0rá”2¨- _wandbruntime‚7185  _runtime‚7185íEýÉ©Ã ' lr-AdamW/pg1‚0.00038474288641489194 ' lr-AdamW/pg2‚0.00038474288641489194 trainer/global_step‚6749 % _timestamp‚1.7884950856013963e+09 _runtime‚7185.241994227 _step‚551¸bû|ÎªÈ ' lr-AdamW/pg1‚0.00038474288641489194 ' lr-AdamW/pg2‚0.00038474288641489194 trainer/global_step‚6749 % _timestamp‚1.7884950856013963e+09 _runtime‚7185.241994227 _step‚551§`xÞ2«- _wandbruntime‚7185  _runtime‚7185²5 ü'¬öO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚478248326097408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6906014132544 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6905243484132 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚572 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 267611216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267634234 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 % _timestamp‚1.7884950856027753e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚6749 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚471318131880448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105158677 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530643917335040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚523417914270720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚478249116745216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚478249116745216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6930194216960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67644117 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105159249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚471318149883392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149639482 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7226203497984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149616466 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6930966861824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149616466 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7226003064320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117994750 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚471318131880448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117994750 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117994752 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149639482 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚466195399738804 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125Žg5©(­ûO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117994750 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚471318131880448 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117994750 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117994752 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149639482 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚466195399738804 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚478248326097408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6906014132544 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6905243484132 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚572 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 267611216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267634234 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 % _timestamp‚1.7884950856027753e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚6749 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚471318131880448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105158677 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530643917335040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚523417914270720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚478249116745216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚478249116745216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6930194216960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67644117 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105159249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚471318149883392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149639482 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7226203497984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149616466 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6930966861824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149616466 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7226003064320¨Œ€s72®- _wandbruntime‚7186  _runtime‚7186AIÓfˆ¯ÿM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149638177 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6907015272630 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105173610 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523493262613504 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118011626 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7186.100281163 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚478317821443584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118011624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267672845 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚575 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6931198962688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚5307203276ÌãTK}08832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚523493286553600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149661219 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 _step‚553 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6931971620864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6931971620864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149638177 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118011624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚478317821443584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚573 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105174185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6906244624010 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267672845 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149661219 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267649801 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530720541483520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6931198962688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149638177 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6907015272630 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105173610 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523493262613504 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118011626 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7186.100281163 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚478317821443584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118011624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267672845 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚575 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6931198962688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530720327608832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚523493286553600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149661219 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 _step‚553 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6931971620864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6931971620864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149638177 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118011624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚478317821443584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚573 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105174185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6906244624010 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267672845 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149661219 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267649801 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530720541483520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6931198962688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0©ðA2±- _wandbruntime‚7186  _runtime‚7186hÅ+V²Ð %loss/total_loss‚41.45623239588114 /loss/FRt3/assignment_loss‚2.757245670775978 0loss/FRt4/assignment_loss‚1.1526077673083446 0loss/SRqqt1/detection_loss‚2.414053629551615 1loss/FBt1/assignment_loss‚0.27042269482788267 1loss/FBt3/assignment_loss‚0.15264391323563586 .loss/FRt4/detection_loss‚1.518695645339665 /loss/FBt4/detection_loss‚0.4300986292376502 2loss/SRqqt3/assignment_loss‚1.3175602198013516 /loss/FRt3/detection_loss‚2.1145845269214134 .loss/FBt3/detection_loss‚2.404679961502552 1loss/SRqqt2/assignment_loss‚2.066555759145154 .loss/FBt2/detection_loss‚2.619499100579156 _runtime‚7186.101926311 epoch‚5 _step‚554 2loss/SRqqt4/assignment_loss‚1.0594290201214775 0loss/SRqqt3/detection_loss‚1.284204061036227 % _timestamp‚1.7884950864617488e+09 /loss/FRt1/assignment_loss‚4.461518408730626 0loss/FRt2/assignment_loss‚3.1862661065282047 2loss/SRqqt1/assignment_loss‚1.6221439491159149 1loss/FBt2/assignment_loss‚0.19110999645636184 1loss/FBt4/assignment_loss‚0.13877649039736925 .loss/FRt2/detection_loss‚2.305810570831307 1loss/SRqqt2/detection_loss‚1.9617188672224681 trainer/global_step‚6749 ,loss/FRt1/detection_loss‚2.5842844521394 1loss/SRqqt4/detection_loss‚1.1039575790797456 .loss/FBt1/detection_loss‚2.338365375995636°ap[³Õ _runtime‚7186.101926311 epoch‚5 _step‚554 2loss/SRqqt4/assignment_loss‚1.0594290201214775 0loss/SRqqt3/detection_loss‚1.284204061036227 % _timestamp‚1.7884950864617488e+09 /loss/FRt1/assignment_loss‚4.461518408730626 0loss/FRt2/assignment_loss‚3.1862661065282047 2loss/SRqqt1/assignment_loss‚1.6221439491159149 1loss/FBt2/assignment_loss‚0.19110999645636184 1loss/FBt4/assignment_loss‚0.13877649039736925 .loss/FRt2/detection_loss‚2.305810570831307 1loss/SRqqt2/detection_loss‚1.9617188672224681 trainer/global_step‚6749 ,loss/FRt1/detection_loss‚2.5842844521394 1loss/SRqqt4/detection_loss‚1.1039575790797456 .loss/FBt1/detection_loss‚2.338365375995636 %loss/total_loss‚41.45623239588114 /loss/FRt3/assignment_loss‚2.757245670775978 0loss/FRt4/assignment_loss‚1.1526077673083446 0loss/SRqqt1/detection_loss‚2.414053629551615 1loss/FBt1/assignment_loss‚0.27042269482788267 1loss/FBt3/assignment_loss‚0.15264391323563586 .loss/FRt4/detection_loss‚1.518695645339665 /loss/FBt4/detection_loss‚0.4300986292376502 2loss/SRqqt3/assignment_loss‚1.3175602198013516 /loss/FRt3/detection_loss‚2.1145845269214134 .loss/FBt3/detection_loss‚2.404679961502552 1loss/SRqqt2/assignment_loss‚2.066555759145154 .loss/FBt2/detection_loss‚2.619499100579156ªjÖ¦´‚(:’ ü‰éÔçÌƯ gpu.0.gpu‚23.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.591' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.918200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù`v‚µ‚(:÷ ü‰éÔ­²»±) disk.nvme0c0n1.out‚174743.03955078125' proc.memory.availableMB‚ 1118.640625* proc.memory.percent‚18.003177642822266 cpu‚25.273885440651704 network.sent‚ 1.6694012e+07) disk./.usagePercent‚78.73011937651803$ disk.dm-0.out‚174743.03955078125 disk.sda.out‚ 531.671875 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0n1.out‚174743.03955078125$ memory_percent‚93.17235946655273 network.recv‚ 2.371811e+06 disk.sda3.in‚0 disk.sda3.out‚ 531.671875 disk.sda2.out‚0" disk.nvme0n1.in‚25365.43359375! proc.memory.rssMB‚ 2949.640625 proc.cpu.threads‚17% disk./.usageGB‚1173.1356735229492 disk.dm-1.out‚ 531.671875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.02734375 disk.sda2.in‚0$ disk.nvme0c0n1.in‚25365.43359375 disk.dm-0.in‚25365.43359375 disk.dm-1.in‚0 disk.sda.in‚06MM†œ¶‚(:‘ ‹ŠéÔÀã—¯ gpu.0.gpu‚38.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚173.658' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.73159999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ûQyˆ·‚(:ý ‹ŠéÔ‡’–³$ memory_percent‚93.16930770874023 cpu‚25.273584292291922 proc.cpu.threads‚17 network.sent‚ 1.6802223e+07 network.recv‚ 2.377915e+06% disk./.usageGB‚1173.1591415405273' disk.nvme0n1.out‚175030.64111328125 disk.dm-1.out‚ 532.04296875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.02734375 disk.sda.in‚0) disk./.usagePercent‚78.73169433486868$ disk.dm-0.out‚175030.64111328125 disk.sda3.out‚ 532.04296875 disk.sda.out‚ 532.04296875 disk.sda1.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚175030.64111328125" disk.nvme0n1.in‚25365.43359375 disk.sda1.out‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 1119.140625! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266$ disk.nvme0c0n1.in‚25365.43359375 disk.dm-0.in‚25365.43359375 disk.sda3.in‚0 disk.dm-1.in‚0¾ÖÄ푸‚(:† šŠéÔ¢õ¾® gpu.0.gpu‚39.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.895' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.579 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä9ñt¹‚(:é šŠéÔç‰ã° disk.dm-0.in‚25408.47265625 disk.dm-1.in‚0 disk.sda1.out‚0 network.recv‚ 2.381331e+06 disk.drbd1260.out‚1.03125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚93.17080974578857! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266) disk./.usagePercent‚78.73269532432755 disk.sda3.in‚0 disk.dm-1.out‚ 532.765625 disk.sda1.in‚0 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚25408.47265625 disk.dm-0.out‚175225.4609375 disk.sda3.out‚ 532.765625 disk.drbd1260.in‚0 disk.sda.out‚ 532.765625) proc.memory.availableMB‚ 1118.89453125 cpu‚25.273176724380793 network.sent‚ 1.6807378e+07$ disk./.usageGB‚1173.174057006836% disk.nvme0c0n1.out‚175225.4609375" disk.nvme0n1.in‚25408.47265625# disk.nvme0n1.out‚175225.46093759«y2º- _wandbruntime‚7236  _runtime‚7236©¨W%Å»¿ & lr-AdamW/pg1‚0.0003790281713391169 & lr-AdamW/pg2‚0.0003790281713391169 trainer/global_step‚6799 $ _timestamp‚1.788495136600576e+09 _runtime‚ 7236.24098692 _step‚555óæ :Ê¼Ä $ _timestamp‚1.788495136600576e+09 _runtime‚ 7236.24098692 _step‚555 & lr-AdamW/pg1‚0.0003790281713391169 & lr-AdamW/pg2‚0.0003790281713391169 trainer/global_step‚6799«Âjsc2½- _wandbruntime‚7236  _runtime‚7236´6ßÿë'¾öO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 % _timestamp‚1.7884951366016223e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6980445917184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚474703128846848 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚527185355350016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150728384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 179462144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118838452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105909467 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚469549579799684 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150705368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37782422 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118838450 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474703146849792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37782950 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6981218562048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6980445917184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481683574764032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6955314069240 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269543818 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚481684365411840 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105908937 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚534464082420224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚481684365411840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚481683574764032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _runtime‚7236.242075591 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150705368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269566836 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 trainer/global_step‚6799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118838450 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚462593495082032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150728384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚469548792111912 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚474703146849792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7278547608064 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocateœE9-d‚ 6956084717652ü(sÎ(¿ûO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚474703128846848 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚527185355350016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150728384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 179462144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118838452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105909467 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚469549579799684 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150705368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37782422 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118838450 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474703146849792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37782950 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6981218562048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6980445917184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481683574764032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6955314069240 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269543818 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚481684365411840 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105908937 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚534464082420224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚481684365411840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚481683574764032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _runtime‚7236.242075591 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150705368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269566836 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 trainer/global_step‚6799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118838450 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚462593495082032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150728384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚469548792111912 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚474703146849792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7278547608064 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6956084717652 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 % _timestamp‚1.7884951366016223e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6980445917184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018¬aŒ‚r2À- _wandbruntime‚7237  _runtime‚7237YK†n'Á€N EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6982223321088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚534540502441472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 _step‚557 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269605447 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474770828786176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚527260703692800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6957085857738 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150727079 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚474770828786176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118855326 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6982223321088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884951375709903e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105924013 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150750121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚462659544478432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150727079 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚527260679752704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 _runtime‚7237.213926895 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118855324 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68136165 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37787850 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6956315209118 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7279608814080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚469615859687550 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68136163 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚474770846789120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7279798748672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118855326 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚481753070110208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269605447 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150750121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825‚Bx 'Â…N BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118855324 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68136165 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37787850 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6956315209118 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7279608814080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚469615859687550 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68136163 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚474770846789120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7279798748672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118855326 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚481753070110208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269605447 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150750121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6982223321088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚534540502441472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 _step‚557 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269605447 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474770828786176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚527260703692800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6957085857738 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150727079 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚474770828786176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118855326 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6982223321088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884951375709903e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105924013 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150750121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚462659544478432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150727079 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚527260679752704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 _runtime‚7237.213926895 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7030697617408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119682152 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478088125813248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚538283828486144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7331288759296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚465896816870032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚472902972172792 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151794270 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478088125813248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚538284061248512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7006155302760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 _step‚560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151794270 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106659926 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚530952748549120 $ _timestamp‚1.788495188621171e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151817286 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚465896799830672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485118823430656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119682152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478088143816192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38051566 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚485119614078464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151817286 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7331079937024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485118823430656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119682152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478088143816192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38051566 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚485119614078464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151817286 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7331079937024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7030697617408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119682152 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478088125813248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚538283828486144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7331288759296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚465896816870032°E<¤l2Ò- _wandbruntime‚7289  _runtime‚7289CW!sÍÓ€N H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119699024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚472970039748638 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151815981 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚478155843755520 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119699026 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151839023 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _step‚561 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 181545984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7032475021312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68618565 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38056879 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119699026 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485187528115712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚538360458033664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119699024 _runtime‚7289.135989802 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151815981 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68618563 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7332337201664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884951894947393e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106674916 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚485188318776832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚472969252060658 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271515005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478155825752576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271538049 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271538049 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005#Ús 'Ô…N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119699024 _runtime‚7289.135989802 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151815981 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68618563 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7332337201664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884951894947393e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106674916 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚485188318776832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚472969252060658 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271515005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478155825752576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271538049 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271538049 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119699024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚472970039748638 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151815981 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚478155843755520 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119699026 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151839023 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _step‚561 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 181545984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7032475021312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68618565 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38056879 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119699026 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485187528115712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚538360458033664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328±¾ýÎv2Õ- _wandbruntime‚7289  _runtime‚7289€˜5[ÖÕ 1loss/SRqqt3/assignment_loss‚1.142126383150325 1loss/FBt1/assignment_loss‚0.11026040573180278 /loss/FRt4/detection_loss‚1.5400790759176015 0loss/SRqqt1/detection_loss‚1.994078447159968 .loss/FRt1/detection_loss‚2.527183109687434 0loss/SRqqt2/detection_loss‚2.335657989537274 1loss/FBt2/assignment_loss‚0.09739552941412583 1loss/FBt3/assignment_loss‚0.48312472051948135 epoch‚5 0loss/FRt2/assignment_loss‚3.1996746448909534 2loss/SRqqt1/assignment_loss‚1.0940228587898768 _runtime‚7289.137856028 1loss/SRqqt3/detection_loss‚1.5476095230117732 /loss/FRt1/assignment_loss‚3.016030102968216 2loss/SRqqt4/assignment_loss‚1.0035593909054221 /loss/FRt3/detection_loss‚2.0660335039315014 /loss/FRt3/assignment_loss‚1.898065122573272 1loss/SRqqt4/detection_loss‚0.9057698911660678 .loss/FBt2/detection_loss‚2.296060534623953 /loss/FRt2/detection_loss‚2.3293937341174624 0loss/FRt4/assignment_loss‚0.8739460027217865 1loss/SRqqt2/assignment_loss‚1.550472969810168 .loss/FBt1/detection_loss‚2.448341652750969 0loss/FBt4/detection_loss‚0.47816584442431725 .loss/FBt3/detection_loss‚2.635143756866455 _step‚562 &loss/total_loss‚37.667103545930985 trainer/global_step‚6849 % _timestamp‚1.7884951894975567e+09 1loss/FBt4/assignment_loss‚0.09490835126078107<å–d`×Ú 1loss/FBt2/assignment_loss‚0.09739552941412583 1loss/FBt3/assignment_loss‚0.48312472051948135 epoch‚5 0loss/FRt2/assignment_loss‚3.1996746448909534 2loss/SRqqt1/assignment_loss‚1.0940228587898768 _runtime‚7289.137856028 1loss/SRqqt3/detection_loss‚1.5476095230117732 /loss/FRt1/assignment_loss‚3.016030102968216 2loss/SRqqt4/assignment_loss‚1.0035593909054221 /loss/FRt3/detection_loss‚2.0660335039315014 /loss/FRt3/assignment_loss‚1.898065122573272 1loss/SRqqt4/detection_loss‚0.9057698911660678 .loss/FBt2/detection_loss‚2.296060534623953 /loss/FRt2/detection_loss‚2.3293937341174624 0loss/FRt4/assignment_loss‚0.8739460027217865 1loss/SRqqt2/assignment_loss‚1.550472969810168 .loss/FBt1/detection_loss‚2.448341652750969 0loss/FBt4/detection_loss‚0.47816584442431725 .loss/FBt3/detection_loss‚2.635143756866455 _step‚562 &loss/total_loss‚37.667103545930985 trainer/global_step‚6849 % _timestamp‚1.7884951894975567e+09 1loss/FBt4/assignment_loss‚0.09490835126078107 1loss/SRqqt3/assignment_loss‚1.142126383150325 1loss/FBt1/assignment_loss‚0.11026040573180278 /loss/FRt4/detection_loss‚1.5400790759176015 0loss/SRqqt1/detection_loss‚1.994078447159968 .loss/FRt1/detection_loss‚2.527183109687434 0loss/SRqqt2/detection_loss‚2.335657989537274²he–Ø‚(:’ ÖŠéÔÑÞ€® gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.922' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.584399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WOU‚Ù‚(:÷ ÖŠéÔ˼æ®) proc.memory.availableMB‚ 1118.34765625* proc.memory.percent‚18.003177642822266 proc.cpu.threads‚17 network.sent‚ 1.6937557e+07) disk./.usagePercent‚78.76834898160612 disk.sda3.in‚0 disk.sda3.out‚ 534.88671875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0$ disk./.usageGB‚1173.705322265625# disk.nvme0c0n1.in‚ 25503.2578125) disk.nvme0c0n1.out‚176580.52978515625 disk.sda1.out‚0" memory_percent‚93.174147605896! proc.memory.rssMB‚ 2949.640625 cpu‚25.27184881741404 disk.dm-1.out‚ 534.88671875 network.recv‚ 2.408219e+06! disk.nvme0n1.in‚ 25503.2578125' disk.nvme0n1.out‚176580.52978515625 disk.drbd1260.out‚ 1.0390625 disk.dm-0.in‚ 25503.2578125$ disk.dm-0.out‚176580.52978515625 disk.sda.in‚0 disk.sda.out‚ 534.88671875 disk.sda2.out‚0 ÈoÀÚ‚(:’ åŠéÔÓÚÕ® gpu.0.gpu‚37.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚174.796' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.959199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Cæ»xÛ‚(:í åŠéÔèÔ²² disk.sda1.out‚0# disk.nvme0c0n1.in‚ 25625.9453125 disk.dm-0.in‚ 25625.9453125 disk.sda3.in‚0 network.recv‚ 2.43434e+06% disk./.usageGB‚1173.7339668273926 disk.drbd1260.in‚0 disk.sda.out‚535.4375 disk.sda1.in‚0! disk.nvme0n1.in‚ 25625.9453125 disk.sda3.out‚535.4375( proc.memory.availableMB‚ 1118.7890625 network.sent‚ 1.7055956e+07 disk.sda2.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚176875.60400390625$ disk.dm-0.out‚176875.60400390625 disk.dm-1.in‚0$ memory_percent‚93.17145347595215! proc.memory.rssMB‚ 2949.640625 disk.drbd1260.out‚ 1.0390625' disk.nvme0n1.out‚176875.60400390625 disk.dm-1.out‚535.4375* proc.memory.percent‚18.003177642822266 cpu‚25.271542103595642 proc.cpu.threads‚17) disk./.usagePercent‚78.77027134218073 disk.sda.in‚0™xI’Ü‚(:‡ ôŠéÔºÉ÷® gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.776' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.9552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºðmCyÝ‚(:î ôŠéÔˆŠ²³ proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚536.0625 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚93.17300319671631* proc.memory.percent‚18.003177642822266$ disk./.usageGB‚1174.235984802246# disk.nvme0c0n1.in‚ 25625.9453125! disk.nvme0n1.in‚ 25625.9453125 disk.sda.out‚536.0625 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.271221072453297 network.sent‚ 1.7063292e+07) disk.nvme0c0n1.out‚177139.37451171875' disk.nvme0n1.out‚177139.37451171875 disk.dm-0.in‚ 25625.9453125 disk.sda3.out‚536.0625 disk.drbd1260.out‚ 1.04296875) proc.memory.availableMB‚ 1118.53515625 network.recv‚ 2.44005e+06) disk./.usagePercent‚78.80396218969429$ disk.dm-0.out‚177139.37451171875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0! proc.memory.rssMB‚ 2949.640625@‘Þ‚(:† ƒ‹éÔ€µœ¯ gpu.0.gpu‚26.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.655' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.531 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*.+…ß‚(:ú ƒ‹éԌ÷³! proc.memory.rssMB‚ 2949.640625 network.sent‚ 1.7068334e+07) disk./.usagePercent‚78.80646005520587% disk./.usageGB‚1174.2732048034668 disk.dm-1.in‚0 disk.sda.out‚ 536.50390625 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 25625.9453125! disk.nvme0n1.in‚ 25625.9453125' disk.nvme0n1.out‚177331.99560546875 disk.dm-0.in‚ 25625.9453125$ disk.dm-0.out‚177331.99560546875' proc.memory.availableMB‚ 1118.796875* proc.memory.percent‚18.003177642822266 cpu‚25.270874565489706 disk.sda3.in‚0 disk.dm-1.out‚ 536.50390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.04296875 disk.sda1.out‚0$ memory_percent‚93.17140579223633 proc.cpu.threads‚17 network.recv‚ 2.443489e+06 disk.sda3.out‚ 536.50390625 disk.sda.in‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚177331.99560546875R€Å2à- _wandbruntime‚7341  _runtime‚7341ÈnûÉáà ' lr-AdamW/pg1‚0.00036761061777674056 ' lr-AdamW/pg2‚0.00036761061777674056 trainer/global_step‚6899 % _timestamp‚1.7884952414507248e+09 _runtime‚7341.091111958 _step‚563££’ÎâÈ ' lr-AdamW/pg2‚0.00036761061777674056 trainer/global_step‚6899 % _timestamp‚1.7884952414507248e+09 _runtime‚7341.091111958 _step‚563 ' lr-AdamW/pg1‚0.00036761061777674056³&c2ã- _wandbruntime‚7341  _runtime‚7341껩Zü'äöO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69091315 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7055455239456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚534720189628416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152883172 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120525850 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120525852 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273432040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚469200138658032 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107410467 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69091317 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7080949317632 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542103804113920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 trainer/global_step‚6899 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488554862745088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152906188 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚469200121618672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273409022 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120525852 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7081721962496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152883172 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107409897 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38318582 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚534720165688320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7383838859264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38319150 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚481473122779648 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542104028487680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7080949317632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _runtime‚7341.092282193 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488554072097280 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7056225887868 % _timestamp‚1.7884952414517927e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152906188 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488554862745088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚481473122779648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚476255576858128 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7383638425600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0fJ°þåûO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488554862745088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152906188 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚469200121618672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273409022 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120525852 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1ñsø#1005 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7081721962496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152883172 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107409897 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38318582 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚534720165688320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7383838859264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38319150 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚481473122779648 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542104028487680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7080949317632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _runtime‚7341.092282193 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488554072097280 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7056225887868 % _timestamp‚1.7884952414517927e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152906188 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488554862745088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚481473122779648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚476255576858128 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7383638425600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69091315 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7055455239456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚534720189628416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152883172 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120525850 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120525852 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273432040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚469200138658032 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107410467 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69091317 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7080949317632 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542103804113920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 trainer/global_step‚6899 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0´ºú²P2æ- _wandbruntime‚7341  _runtime‚73412Çæ…'çÿM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273470651 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚542180416798720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7081954063360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38324335 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152904883 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7082726721536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7082726721536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚469266188054432 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚542180188243968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7057227027954 trainer/global_step‚6899 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7081954063360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69100965 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚469266205093792 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152904883 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7384674212864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚555 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152927925 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273470651 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7341.939078289 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚488622776782336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273447607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚481540840721920 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120542726 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107425300 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476323432121746 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚481540840721920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚488623567443456 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107424745 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 _step‚565 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120542726 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120542724 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚481540822718976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚534795537971200 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚481540822718976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚534795537971200 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚555 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152927925 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273470651 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7341.939078289 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚488622776782336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273447607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚481540840721920 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120542726 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107425300 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476323432121746 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚481540840721920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚488623567443456 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107424745 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 _step‚565 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120542726 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825µ_V?ë2é- _wandbruntime‚7341  _runtime‚7341ˆÖhaêÛ /loss/FBt3/detection_loss‚2.6119757095972695 % _timestamp‚1.7884952423005986e+09 2loss/SRqqt1/assignment_loss‚1.4293850918766111 1loss/FBt4/assignment_loss‚0.07748556691522823 /loss/FRt1/detection_loss‚2.3064333519492393 /loss/FRt2/detection_loss‚2.5533628492744125 1loss/SRqqt1/detection_loss‚2.2168055195361376 .loss/FBt1/detection_loss‚3.234746253490448 1loss/FBt2/assignment_loss‚0.14232643508876208 1loss/SRqqt3/detection_loss‚1.6436561603867819 /loss/FBt4/detection_loss‚0.6761618253393945 0loss/FRt4/assignment_loss‚1.4841842201520812 _step‚566 0loss/FRt1/assignment_loss‚3.0717085034419327 2loss/SRqqt4/assignment_loss‚1.0622725989059576 1loss/FBt3/assignment_loss‚0.14831043741052477 .loss/FRt3/detection_loss‚2.406311572362215 &loss/total_loss‚41.705594716543686 epoch‚5 0loss/FBt1/assignment_loss‚0.3039156001061201 /loss/FRt4/detection_loss‚1.7767152964787662 0loss/FRt2/assignment_loss‚3.2034897468984127 2loss/SRqqt2/assignment_loss‚2.0041514246360115 1loss/SRqqt4/detection_loss‚0.9244933207455563 trainer/global_step‚6899 0loss/FRt3/assignment_loss‚2.3874244766357617 2loss/SRqqt3/assignment_loss‚1.1460023475663057 1loss/SRqqt2/detection_loss‚2.8933897588564004 /loss/FBt2/detection_loss‚2.0008866488933563 _runtime‚7341.940764546@.>fëà 2loss/SRqqt3/assignment_loss‚1.1460023475663057 1loss/SRqqt2/detection_loss‚2.8933897588564004 /loss/FBt2/detection_loss‚2.0008866488933563 _runtime‚7341.940764546 /loss/FBt3/detection_loss‚2.6119757095972695 % _timestamp‚1.7884952423005986e+09 2loss/SRqqt1/assignment_loss‚1.4293850918766111 1loss/FBt4/assignment_loss‚0.07748556691522823 /loss/FRt1/detection_loss‚2.3064333519492393 /loss/FRt2/detection_loss‚2.5533628492744125 1loss/SRqqt1/detection_loss‚2.2168055195361376 .loss/FBt1/detection_loss‚3.234746253490448 1loss/FBt2/assignment_loss‚0.14232643508876208 1loss/SRqqt3/detection_loss‚1.6436561603867819 /loss/FBt4/detection_loss‚0.6761618253393945 0loss/FRt4/assignment_loss‚1.4841842201520812 _step‚566 0loss/FRt1/assignment_loss‚3.0717085034419327 2loss/SRqqt4/assignment_loss‚1.0622725989059576 1loss/FBt3/assignment_loss‚0.14831043741052477 .loss/FRt3/detection_loss‚2.406311572362215 &loss/total_loss‚41.705594716543686 epoch‚5 0loss/FBt1/assignment_loss‚0.3039156001061201 /loss/FRt4/detection_loss‚1.7767152964787662 0loss/FRt2/assignment_loss‚3.2034897468984127 2loss/SRqqt2/assignment_loss‚2.0041514246360115 1loss/SRqqt4/detection_loss‚0.9244933207455563 trainer/global_step‚6899 0loss/FRt3/assignment_loss‚2.3874244766357617¶œªùì‚(:’ ’‹éÔ¹‡Ã¯ gpu.0.gpu‚41.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.876' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.975200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò˜7çí‚(:î ’‹éÔեܰ* proc.memory.percent‚18.003177642822266) disk./.usagePercent‚78.80644213467848( disk.nvme0c0n1.out‚177780.8994140625 disk.dm-0.in‚ 25625.9453125# disk.dm-0.out‚177780.8994140625 disk.drbd1260.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 network.sent‚ 1.7073207e+07 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 537.078125 disk.dm-1.in‚0 disk.dm-1.out‚ 537.078125 disk.drbd1260.out‚1.046875$ memory_puvî~’ercent‚93.17159652709961# disk.nvme0c0n1.in‚ 25625.9453125! disk.nvme0n1.in‚ 25625.9453125& disk.nvme0n1.out‚177780.8994140625! proc.memory.rssMB‚ 2949.640625 cpu‚25.270548478394907 network.recv‚ 2.44652e+06% disk./.usageGB‚1174.2729377746582 disk.sda2.out‚0 disk.sda.out‚ 537.078125 disk.sda1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1118.765625p;ó|î‚(:’ ¡‹éÔÄí­ gpu.0.gpu‚45.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.373' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.674599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þœ¢žƒï‚(:ø ¡‹éÔÌ›À± disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2949.640625( disk.nvme0c0n1.out‚178205.4306640625 disk.dm-1.in‚0 disk.sda1.in‚0# memory_percent‚93.1718111038208* proc.memory.percent‚18.003177642822266 proc.cpu.threads‚17 network.recv‚ 2.45999e+06% disk./.usageGB‚1174.3354377746582& disk.nvme0n1.out‚178205.4306640625 disk.dm-0.in‚25786.08203125 disk.sda3.in‚0 disk.sda3.out‚ 537.51171875 disk.drbd1260.out‚1.046875 disk.sda2.in‚0 network.sent‚ 1.7185539e+07) disk./.usagePercent‚78.81063656211947# disk.dm-0.out‚178205.4306640625 disk.dm-1.out‚ 537.51171875 disk.drbd1260.in‚0 disk.sda.out‚ 537.51171875) proc.memory.availableMB‚ 1118.73046875 cpu‚25.270266646876898$ disk.nvme0c0n1.in‚25786.08203125" disk.nvme0n1.in‚25786.08203125„Ì£ð‚(:’ °‹éÔ¡óë® gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.226' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.245200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐàÔdwñ‚(:ì °‹éÔÉ÷à± disk.sda2.in‚0" disk.nvme0n1.in‚25786.08203125( proc.memory.availableMB‚ 1116.9765625 network.recv‚ 2.483636e+06) disk./.usagePercent‚78.81902541700146% disk./.usageGB‚1174.4604377746582 disk.drbd1260.out‚ 1.05078125$ disk.nvme0c0n1.out‚ 178654.328125" disk.nvme0n1.out‚ 178654.328125! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 537.9296875$ disk.nvme0c0n1.in‚25786.08203125 disk.dm-0.in‚25786.08203125 disk.dm-0.out‚ 178654.328125 cpu‚25.269922498410246 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 537.9296875 disk.dm-1.out‚ 537.9296875 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚93.18251609802246 network.sent‚ 1.7201044e+07 disk.sda.in‚0 disk.sda1.in‚0öwA¡2ò- _wandbruntime‚7393  _runtime‚7393ËÁƳÆóÀ & lr-AdamW/pg1‚0.0003619101586911886 & lr-AdamW/pg2‚0.0003619101586911886 trainer/global_step‚6949 % _timestamp‚1.7884952938550663e+09 _runtime‚ 7393.49534231 _step‚567êÖ´ËôÅ _step‚567 & lr-AdamW/pg1‚0.0003619101586911886 & lr-AdamW/pg2‚0.0003619101586911886 trainer/global_step‚6949 % _timestamp‚1.7884952938550663e+09 _runtime‚ 7393.49534231· ¶+»2õ- _wandbruntime‚7393  _runtime‚7393âíY´ü'ööO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153972074 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7131201017856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7436251481088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491989320763904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484858119746048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7131973662720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚564 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38586175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚479609756919008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108160456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7131973662720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108159890 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚538487606767616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 _step‚568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7106296472976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121369550 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153972074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚472503460446032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69573715 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6949 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491989320763904 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚545923858248704 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491990111411712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7436046853120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚472503443406672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 275341624 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7131201017856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38586739 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚545923629680640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484858119746048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484858137748992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 _runtime‚7393.496410974 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚538487582827520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 275364642 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69573717 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484858137748992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 % _timestamp‚1.7884952938559856e+09†us(÷ûO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚538487606767616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 _step‚568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7106296472976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121369550 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153972074 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚472503460446032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69573715 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚6949 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491989320763904 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚545923858248704 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491990111411712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7436046853120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚472503443406672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 275341624 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7131201017856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38586739 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚545923629680640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484858119746048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484858137748992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 _runtime‚7393.496410974 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚538487582827520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 275364642 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69573717 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484858137748992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 % _timestamp‚1.7884952938559856e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153972074 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7131201017856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7436251481088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491989320763904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484858119746048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7131973662720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚564 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38586175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚479609756919008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108160456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7131973662720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108159890¸¿qGº2ø- _wandbruntime‚7394  _runtime‚7394þÍh»!ùÿM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69583363 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154016827 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚472569509842432 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275403253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7132978421760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7132978421760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚538562931170304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7106526964442 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚546000251590656 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154016827 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153993785 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108174888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69583365 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚569 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121386424 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚479676824494854 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚538562955110400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 trainer/global_step‚6949 $ _timestamp‚1.788495294704118e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚566 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚479676036806874 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121386426 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7437106545664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153993785 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275380209 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7132205763584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚492058816110080 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚492058025448960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7132205763584 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108175456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275403253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚546000037715968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚568 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _runtime‚7394.344906065 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121386424 D>DeviceStatsMonitor.on_train_batch_end/segment.l¤~ÀJarge_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275380209 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7107297613062 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484925819685376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7437296480256 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚546000251590656 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154016827 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153993785 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108174888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69583365 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚569 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121386424 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚479676824494854 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚538562955110400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 trainer/global_step‚6949 $ _timestamp‚1.788495294704118e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚566 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚479676036806874 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121386426 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7437106545664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153993785 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275380209 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7132205763584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚492058816110080 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚492058025448960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7132205763584 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108175456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275403253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚546000037715968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚568 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _runtime‚7394.344906065 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121386424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275380209 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7107297613062 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484925819685376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7437296480256 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚472569509842432 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275403253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7132978421760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7132978421760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚538562931170304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7106526964442 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚475806782234032 _runtime‚7442.432320011 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7182225362944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155060976 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 277274226 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7181452718080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122213252 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚482963149292116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7181452718080 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚475806765194672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7156367058084 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚495425360078336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚542254999966720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108910592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚482962361604344 % _timestamp‚1.7884953427918143e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155083992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70056117 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 trainer/global_step‚6999 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚542255023906816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚572 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7182225362944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 277297244 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚549744097661440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122213250 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155060976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚495424569430528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7489073754624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122213252 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚495425360078336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488243134715392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚488243116712448 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108911147 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38855030 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚549743879579136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚495424569430528 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122213250 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155083992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489ƒ+°Q ‹ûO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 trainer/global_step‚6999 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚542255023906816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚572 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7182225362944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 277297244 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚549744097661440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122213250 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_trainÃKÉ?ù_batch_start/allocation.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155060976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚495424569430528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7489073754624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122213252 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚495425360078336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488243134715392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚488243116712448 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108911147 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38855030 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚549743879579136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚495424569430528 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122213250 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155083992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚488243116712448 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚475806782234032 _runtime‚7442.432320011 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7182225362944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155060976 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 277274226 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7181452718080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122213252 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚482963149292116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7181452718080 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚475806765194672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7156367058084 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚495425360078336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚542254999966720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108910592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚482962361604344 % _timestamp‚1.7884953427918143e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155083992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70056117¼öÙ¨2Œ- _wandbruntime‚7443  _runtime‚7443œ/'€N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚475872831630432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚488310816651776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277335855 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7182457463808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚495493274115584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚488310816651776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7183230121984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7156597549550 _runtime‚7443.415397495 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 277335855 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155082687 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 277312811 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚549820502750720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488310834654720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚6999 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108926112 _step‚573 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7489940566528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155082687 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108925569 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7157368198170 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚488310834654720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541«yÛ 'Ž…N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚488310834654720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚475872831630432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚488310816651776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277335855 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7182457463808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚495493274115584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚488310816651776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7183230121984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7156597549550 _runtime‚7443.415397495 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 277335855 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155082687 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 277312811 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚549820502750720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488310834654720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚6999 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108926112 _step‚573 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7489940566528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155082687 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108925569 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7157368198170½‚®ˆâ2- _wandbruntime‚7443  _runtime‚7443 _Ù 0loss/FRt3/assignment_loss‚3.0489392002423603 2loss/SRqqt1/assignment_loss‚1.5533778355629355 1loss/SRqqt3/assignment_loss‚1.527211621443325 /loss/FBt3/detection_loss‚2.0980518087744713 .loss/FBt1/detection_loss‚2.299353265762329 1loss/FBt3/assignment_loss‚0.36960342402562674 .loss/FRt1/detection_loss‚2.524497180538518 1loss/SRqqt1/detection_loss‚2.2703495268468505 epoch‚5 _step‚574 0loss/FRt1/assignment_loss‚3.1660999843052458 0loss/FRt4/assignment_loss‚1.2284645448470939 2loss/SRqqt4/assignment_loss‚1.0191742123049847 2loss/FBt1/assignment_loss‚0.040514271949496104 1loss/FBt2/assignment_loss‚0.06336414508694474 .loss/FRt3/detection_loss‚2.611681344774034 /loss/FRt4/detection_loss‚1.6185355807879362 /loss/FRt2/detection_loss‚2.6162188317820303 1loss/SRqqt4/detection_loss‚0.7922923287924598 .loss/FBt2/detection_loss‚2.434331274032593 &loss/total_loss‚40.354091690706596 % _timestamp‚1.7884953437784402e+09 1loss/FBt4/assignment_loss‚0.04788523074216268 0loss/FRt2/assignment_loss‚2.6014123453813442 2loss/SRqqt2/assignment_loss‚1.5806856390088797 1loss/SRqqt2/detection_loss‚2.2505367946624757 0loss/SRqqt3/detection_loss‚1.945460725971993 /loss/FBt4/detection_loss‚0.6460505730805121 trainer/global_step‚6999 _runtime‚7443.419183839xâvd‘Þ trainer/global_step‚6999 _runtime‚7443.419183839 0loss/FRt3/assignment_loss‚3.0489392002423603 2loss/SRqqt1/assignment_loss‚1.5533778355629355 1loss/SRqqt3/assignment_loss‚1.527211621443325 /loss/FBt3/detection_loss‚2.0980518087744713 .loss/FBt1/detection_loss‚2.299353265762329 1loss/FBt3/assignment_loss‚0.36960342402562674 .loss/FRt1/detection_loss‚2.524497180538518 1loss/SRqqt1/detection_loss‚2.2703495268468505 epoch‚5 _step‚574 0loss/FRt1/assignment_loss‚3.1660999843052458 0loss/FRt4/assignment_loss‚1.2284645448470939 2loss/SRqqt4/assignment_loss‚1.0191742123049847 2loss/FBt1/assignment_loss‚0.040514271949496104 1loss/FBt2/assignment_loss‚0.06336414508694474 .loss/FRt3/detection_loss‚2.611681344774034 /loss/FRt4/detection_loss‚1.6185355807879362 /loss/FRt2/detection_loss‚2.6162188317820303 1loss/SRqqt4/detection_loss‚0.7922923287924598 .loss/FBt2/detection_loss‚2.434331274032593 &loss/total_loss‚40.354091690706596 % _timestamp‚1.7884953437784402e+09 1loss/FBt4/assignment_loss‚0.04788523074216268 0loss/FRt2/assignment_loss‚2.6014123453813442 2loss/SRqqt2/assignment_loss‚1.5806856390088797 1loss/SRqqt2/detection_loss‚2.2505367946624757 0loss/SRqqt3/detection_loss‚1.945460725971993 /loss/FBt4/detection_loss‚0.6460505730805121¾ã0¦j’’‚(:‡ û‹éÔØ®ÿ° gpu.0.gpu‚36.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.427' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î%½“y“‚(:î û‹éÔÄ앲 disk.drbd1260.out‚ 1.05859375$ disk.nvme0c0n1.in‚25897.50390625 disk.sda2.out‚0' disk.nvme0c0n1.out‚180241.302734375" disk.nvme0n1.in‚25897.50390625% disk.nvme0n1.out‚180241.302734375" disk.dm-0.out‚180241.302734375) proc.memory.availableMB‚ 1118.39453125! proc.memory.rssMB‚ 2949.640625( disk./.usagePercent‚78.8389394750674 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚25897.50390625* proc.memory.percent‚18.003177642822266 network.recv‚ 2.505525e+06 disk.sda.in‚0 disk.sda.out‚ 541.03125 disk.sda1.out‚0$ memory_percent‚93.17386150360107 cpu‚25.26832784117092 proc.cpu.threads‚17 network.sent‚ 1.7331485e+07% disk./.usageGB‚1174.7571716308594 disk.sda3.in‚0 disk.sda3.out‚ 541.03125 disk.dm-1.out‚ 541.03125 disk.drbd1260.in‚0d›  n”‚(:ã ŠŒéÔŸ†š¯) disk./.usagePercent‚78.83893973107494 disk.sda1.out‚0 disk.dm-0.in‚25897.50390625 disk.drbd1260.out‚1.0625$ memory_percent‚93.17541122436523! proc.memory.rssMB‚ 2949.640625 cpu‚25.26804722970004 proc.cpu.threads‚17 network.sent‚ 1.7439288e+07" disk.nvme0n1.in‚25897.50390625" disk.dm-0.out‚180602.330078125 disk.sda3.in‚0 disk.sda3.out‚541.625 disk.drbd1260.in‚0' proc.memory.availableMB‚ 1118.140625 network.recv‚ 2.510843e+06 disk.sda2.in‚0$ disk.nvme0c0n1.in‚25897.50390625 disk.dm-1.out‚541.625* proc.memory.percent‚18.003177642822266% disk./.usageGB‚1174.7571754455566 disk.sda.in‚0 disk.sda.out‚541.625 disk.sda1.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚180602.330078125% disk.nvme0n1.out‚180602.330078125 disk.dm-1.in‚0¢ç ‡ •‚(:Y… ‰‡ ŠŒéÔ¾„‹± gpu.0.gpu‚33.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.7348 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uL¹w’–‚(:‡ ™ŒéÔ¯…›¯ gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚125.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1172 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04¥ªÓf—‚(:Û ™ŒéÔˆÀ´³ disk.sda2.in‚0 disk.nvme0n1.in‚ 26034.015625 proc.cpu.threads‚17 disk.dm-1.out‚542.25 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 26034.015625% disk.nvme0n1.out‚180833.607421875' proc.memory.availableMB‚ 1118.109375* proc.memory.percent‚18.003177642822266 disk.drbd1260.out‚1.0625 disk.sda.out‚542.25 network.sent‚ 1.7445588e+07% disk./.usageGB‚1174.7341957092285 disk.dm-0.in‚ 26034.015625 disk.sda3.out‚542.25 disk.drbd1260.in‚0 disk.sda1.out‚0) disk./.usagePercent‚78.83739754168869 disk.dm-1.in‚0' disk.nvme0c0n1.out‚180833.607421875$ memory_percent‚93.17560195922852! proc.memory.rssMB‚ 2949.640625 cpu‚25.267751056517252 network.recv‚ 2.515686e+06" disk.dm-0.out‚180833.607421875 disk.sda3.in‚0*p2˜- _wandbruntime‚7494  _runtime‚7494Ýab·Ç™Á & lr-AdamW/pg1‚0.0003505318154994336 & lr-AdamW/pg2‚0.0003505318154994336 trainer/global_step‚7049 % _timestamp‚1.7884953951762373e+09 _runtime‚7494.816503056 _step‚575É(äÌšÆ _step‚575 & lr-AdamW/pg1‚0.0003505318154994336 & lr-AdamW/pg2‚0.0003505318154994336 trainer/global_step‚7049 % _timestamp‚1.7884953951762373e+09 _runtime‚7494.816503056¿^›H2›- _wandbruntime‚7494  _runtime‚7494N Áaü'œöO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39122467 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39121942 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚553564085762560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156172894 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 % _timestamp‚1.7884953951771615e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚479110104022032 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109660984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 181559296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123056952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546022441046016 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123056952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚553563880263168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚491628131681792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279229846 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚491628113678848 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚491628113678848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚498860608744960 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109660457 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123056950 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7232477063168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚498859818097152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156149878 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚498860608744960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚546022417105920 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205499392 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7205666994780 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7231704418304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527IkFì(ûO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚553563880263168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚491628131681792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279229846 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚491628113678848 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚491628113678848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚498860608744960 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109660457 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123056950 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7232477063168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚498859818097152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156149878 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚498860608744960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚546022417105920 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205499392 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7205666994780 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7231704418304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39122467 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39121942 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚553564085762560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156172894 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 % _timestamp‚1.7884953951771615e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚479110104022032 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109660984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 181559296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123056952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546022441046016 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123056952À•P…2ž- _wandbruntime‚7495  _runtime‚7495ÿçR7'Ÿ€N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156194631 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7233481822208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 279268457 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚553640502296064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚491695813618176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 trainer/global_step‚7049 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 279245413 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7542522972672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚7495.848589276 _step‚577 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123073824 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚553640288421376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109675507 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7542712907264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚479176153418432 % _timestamp‚1.7884953962060626e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279268457 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚479176170457792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚491695831621120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123073826 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156171589 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70548163 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39127880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7233481822208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7207438783278 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚491695813618176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚486382821553090 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚498929313443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚498929313443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 279245413 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156194631 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚491695831621120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328qÒ€ ' …N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156194631 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7233481822208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 279268457 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚553640502296064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚491695813618176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 trainer/global_step‚7049 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 279245413 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7542522972672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚7495.848589276 _step‚577 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123073824 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚553640288421376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109675507 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7542712907264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚479176153418432 % _timestamp‚1.7884953962060626e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279268457 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚479176170457792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚491695831621120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123073826 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156171589 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70548163 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39127880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7233481822208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7207438783278 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚491695813618176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚486382821553090 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚498929313443328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚498929313443328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 279245413 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156194631 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚491695831621120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328Á1[lL2¡- _wandbruntime‚7495  _runtime‚7495Hé’Z¢Ô trainer/global_step‚7049 0loss/FRt2/assignment_loss‚3.1347340204213796 /loss/FRt3/detection_loss‚2.7451497548156314 /loss/FBt1/detection_loss‚2.0473694735103183 $loss/total_loss‚40.3410622653491 2loss/SRqqt1/assignment_loss‚1.6698081405109926 /loss/FBt3/detection_loss‚2.1504400730133058 /loss/FBt4/detection_loss‚0.5330677149415185 0loss/SRqqt3/assignment_loss‚1.47123700643287 0loss/SRqqt3/detection_loss‚1.863434629405246 % _timestamp‚1.7884953962110968e+09 /loss/FRt4/assignment_loss‚1.053619984537363 2loss/SRqqt2/assignment_loss‚1.2318328561621807 1loss/FBt1/assignment_loss‚0.41026426474046374 1loss/FBt2/assignment_loss‚0.25093318516976687 .loss/FRt4/detection_loss‚1.506555761792697 _runtime‚7495.851719214 2loss/SRqqt4/assignment_loss‚1.0715185058856938 0loss/SRqqt1/detection_loss‚2.415927001408168 1loss/SRqqt4/detection_loss‚1.0623747380198658 epoch‚5 1loss/FBt3/assignment_loss‚0.20237131822068477 1loss/FBt4/assignment_loss‚0.11567817612499526 /loss/FRt1/detection_loss‚2.2157220685990846 0loss/SRqqt2/detection_loss‚2.321199001804475 .loss/FBt2/detection_loss‚2.101342494998659 0loss/FRt1/assignment_loss‚3.7727426959247126 0loss/FRt3/assignment_loss‚2.5382198045651116 _step‚578 /loss/FRt2/detection_loss‚2.4555195943439174 +{R_£Ù /loss/FRt4/assignment_loss‚1.053619984537363 2loss/SRqqt2/assignment_loss‚1.2318328561621807 1loss/FBt1/assignment_loss‚0.41026426474046374 1loss/FBt2/assignment_loss‚0.25093318516976687 .loss/FRt4/detection_loss‚1.506555761792697 _runtime‚7495.851719214 2loss/SRqqt4/assignment_loss‚1.0715185058856938 0loss/SRqqt1/detection_loss‚2.415927001408168 1loss/SRqqt4/detection_loss‚1.0623747380198658 epoch‚5 1loss/FBt3/assignment_loss‚0.20237131822068477 1loss/FBt4/assignment_loss‚0.11567817612499526 /loss/FRt1/detection_loss‚2.2157220685990846 0loss/SRqqt2/detection_loss‚2.321199001804475 .loss/FBt2/detection_loss‚2.101342494998659 0loss/FRt1/assignment_loss‚3.7727426959247126 0loss/FRt3/assignment_loss‚2.5382198045651116 _step‚578 /loss/FRt2/detection_loss‚2.4555195943439174 trainer/global_step‚7049 0loss/FRt2/assignment_loss‚3.1347340204213796 /loss/FRt3/detection_loss‚2.7451497548156314 /loss/FBt1/detection_loss‚2.0473694735103183 $loss/total_loss‚40.3410622653491 2loss/SRqqt1/assignment_loss‚1.6698081405109926 /loss/FBt3/detection_loss‚2.1504400730133058 /loss/FBt4/detection_loss‚0.5330677149415185 0loss/SRqqt3/assignment_loss‚1.47123700643287 0loss/SRqqt3/detection_loss‚1.863434629405246 % _timestamp‚1.7884953962110968e+09¦0­›¤‚(:’ ¨ŒéÔ†öñ° gpu.0.gpu‚49.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.478' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.695600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01âFíl¥‚(:á ¨ŒéÔéÉʳ disk.drbd1260.in‚0 disk.sda.out‚542.875 disk.sda2.out‚0$ disk.nvme0n1.out‚181210.69140625 disk.dm-0.in‚ 26034.015625! proc.memory.rssMB‚ 2949.640625 network.sent‚ 1.7451433e+07 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.06640625 disk.sda1.out‚0! disk.dm-0.out‚181210.69140625$ memory_percent‚93.17572116851807) proc.memory.availableMB‚ 1118.08984375 cpu‚25.267450578369207% disk./.usageGB‚1174.7577209472656 disk.sda1.in‚0& disk.nvme0c0n1.out‚181210.69140625* proc.memory.percent‚18.003177642822266 disk.sda3.in‚0 disk.sda3.out‚542.875 disk.dm-1.out‚542.875 disk.sda.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 26034.015625 disk.nvme0n1.in‚ 26034.015625 proc.cpu.threads‚17 network.recv‚ 2.520017e+06) disk./.usagePercent‚78.83897634015233Xa?ᦂ(:’ ·ŒéÔü¸í¬ gpu.0.gpu‚22.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.372' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.674400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'µ˜ƒc§‚(:Ø ·ŒéÔ™ŠÎ¯ disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 26034.015625 disk.nvme0n1.in‚ 26034.015625 disk.dm-1.out‚543.5! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 network.sent‚ 1.7559577e+07 disk.dm-0.in‚ 26034.015625! disk.dm-0.out‚181451.35546875$ memory_percent‚93.17584037780762( proc.memory.availableMB‚ 1118.0703125 cpu‚25.2671145487409 proc.cpu.threads‚17 network.recv‚ 2.525943e+06 disk.sda2.in‚0 disk.sda3.out‚543.5 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.06640625 disk.sda.in‚0 disk.sda.out‚543.5 disk.sda1.in‚0) disk./.usagePercent‚78.83897634015233% disk./.usageGB‚1174.7577209472656 disk.sda2.out‚0& disk.nvme0c0n1.out‚181451.35546875$ disk.nvme0n1.out‚181451.35546875Üß`䨂(:’ ÆŒéÔá¨ÿ¯ gpu.0.gpu‚41.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚140.993' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.198600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ¢ÙΩ‚(:ö ÆŒéԛꇳ network.sent‚ 1.7564604e+07 disk.sda2.in‚0 disk.nvme0n1.in‚ 26034.015625 disk.drbd1260.out‚ 1.0703125 disk.sda.out‚ 544.10546875 cpu‚25.26679437036198 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 26034.015625) disk.nvme0c0n1.out‚181829.05712890625$ disk.dm-0.out‚181829.05712890625 disk.sda3.in‚0 disk.sda3.out‚ 544.10546875 disk.dm-1.in‚0 network.recv‚ 2.529545e+06) disk./.usagePercent‚78.87097088174401$ disk./.usageGB‚1175.234462738037' disk.nvme0n1.out‚181829.05712890625 disk.dm-0.in‚ 26034.015625 disk.dm-1.out‚ 544.10546875 disk.drbd1260.in‚0$ memory_percent‚93.17429065704346 disk.sda2.out‚0 disk.sda.in‚0) proc.memory.availableMB‚ 1118.32421875! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 proc.cpu.threads‚17,&’ª‚(:‡ ÕŒéÔ‹øô® gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.831' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ[øõ€«‚(:õ ÕŒéÔ¾õš³ disk.sda2.out‚0! disk.nvme0c0n1.in‚ 26047.59375 disk.dm-0.in‚ 26047.59375$ disk.dm-0.out‚182044.52197265625! proc.memory.rssMB‚ 2949.640625 proc.cpu.threads‚17) disk./.usagePercent‚78.87254584009466 disk.sda3.out‚ 545.10546875 disk.dm-1.in‚0 disk.sda.in‚0$ memory_percent‚93.17429065704346 cpu‚25.266530881770834 network.sent‚ 1.7572647e+07 network.recv‚ 2.534214e+06 disk.drbd1260.out‚ 1.0703125 disk.sda1.out‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚182044.52197265625) proc.memory.availableMB‚ 1118.32421875* proc.memory.percent‚18.003177642822266 disk.sda1.in‚0% disk./.usageGB‚1175.2579307556152 disk.sda3.in‚0 disk.dm-1.out‚ 545.10546875 disk.sda.out‚ 545.10546875 disk.nvme0n1.in‚ 26047.59375' disk.nvme0n1.out‚182044.52197265625 disk.drbd1260.in‚0ZQµ92¬- _wandbruntime‚7545  _runtime‚7545ÊÓÀ~Ç­Á % _timestamp‚1.7884954460383277e+09 _runtime‚7545.678927072 _step‚579 & lr-AdamW/pg1‚0.0003448563026228975 & lr-AdamW/pg2‚0.0003448563026228975 trainer/global_step‚7099Sz™Ì®Æ % _timestamp‚1.7884954460383277e+09 _runtime‚7545.678927072 _step‚579 & lr-AdamW/pg1‚0.0003448563026228975 & lr-AdamW/pg2‚0.0003448563026228975 trainer/global_step‚7099Ã.&,s2¯- _wandbruntime‚7545  _runtime‚75459âXû'°õO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489669146350560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7281956118528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7282728763392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123900650 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚495013128648192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39391316 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7594293055488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281139430 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157238780 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110411688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489669934038332 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482413408770672 $ _timestamp‚1.788495446039907e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110412233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚549789834245120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157261796 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502295066763776 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚502295857411584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚502295066763776 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209693696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚549789858185216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123900652 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚495013110645248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7255737579888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7099 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157261796 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123900650 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71020917 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _runtime‚7545.680631073 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557384127300608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185753600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚495013128648192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7282728763392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123900652 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _step‚580 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚502295857411584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005°Va±úO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281162448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚482413425810032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚495013110645248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7255737579888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7099 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157261796 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123900650 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71020917 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _runtime‚7545.680631073 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557384127300608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185753600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚495013128648192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7282728763392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123900652 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _step‚580 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚502295857411584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489669146350560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7281956118528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_starH”]Ÿt/active.small_pool.current‚23016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7282728763392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123900650 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚495013128648192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39391316 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7594293055488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281139430 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157238780 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110411688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489669934038332 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482413408770672 $ _timestamp‚1.788495446039907e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110412233 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚549789834245120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157261796 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502295066763776 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚502295857411584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚502295066763776 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209693696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚549789858185216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123900652 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281178015 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7257509368386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7282960864256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7283733522432 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110426689 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281201059 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489737001614178 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 $ _timestamp‚1.788495447110405e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7282960864256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157283533 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚495080828587520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7595345227264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482479492245792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157283533 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123917524 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281201059 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39396672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚495080810584576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39396126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157260491 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281178015 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7257509368386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7282960864256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7283733522432 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110426689 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281201059 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489737001614178 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 $ _timestamp‚1.788495447110405e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7282960864256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157283533 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚495080828587520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7595345227264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482479492245792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157283533 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123917524 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281201059 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39396672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚495080810584576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39396126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157260491 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 283095050 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158350698 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚498398107611648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚561204291038208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚485716730558672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158327682 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158327682 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505731106078208 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚498398107611648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7332207818752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213888000 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39658863 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 trainer/global_step‚7149 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505730315430400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7332207818752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7332980463616 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124744352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124744350 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚493023326411440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71503315 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553557275324416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553557251384320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚498398125614592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71503317 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 283095050 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158350698 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚498398107611648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚561204291038208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚485716730558672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158327682 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158327682 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505731106078208 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚498398107611648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7332207818752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213888000 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39658863 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 trainer/global_step‚7149 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505730315430400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7332207818752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7332980463616 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124744352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124744350 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚493023326411440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71503315 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553557275324416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553557251384320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚498398125614592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71503317 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111177233 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 283110617 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283133661 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 _step‚585 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚498465825553920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚485782814033792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 283110617 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71512963 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7333212564480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚505799020115456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39664270 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158349393 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7333985222656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚561280695341056 trainer/global_step‚7149 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124761224 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158372435 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚551 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚553632623667200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505799810776576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553632599727104 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 % _timestamp‚1.7884954993089497e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚493089606299306 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7333212564480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71512965 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚493090393987286 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7306809304874 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚493089606299306 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7333212564480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71512965 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 WCDeviceStatsMonitor.on_train_batch_end/requestõG”ç!ed_bytes.all.allocated‚493090393987286 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7306809304874 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111177233 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 283110617 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283133661 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 _step‚585 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚498465825553920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚485782814033792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 283110617 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71512963 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7333212564480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚505799020115456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39664270 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158349393 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7333985222656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚561280695341056 trainer/global_step‚7149 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124761224 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158372435 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚551 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚553632623667200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505799810776576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553632599727104 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 % _timestamp‚1.7884954993089497e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7699582241280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚557324692463616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501783122580992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚501783104578048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚565024470944256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125588050 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚489020052346672 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7355878750104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111912758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71985715 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚557324668523520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚509166354744832 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125588052 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71985717 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159439600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚496376718784548 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159416584 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚509165564097024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚565024250764800 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚496375931096776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7382459518976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚565024470944256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125588050 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚489020052346672 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7355878750104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111912758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71985715 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚557324668523520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚509166354744832 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125588052 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71985717 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159439600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚496376718784548 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159416584 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚509165564097024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚565024250764800 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚496375931096776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7382459518976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7699582241280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚557324692463616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489ÌÛÓ2Ö- _wandbruntime‚7648  _runtime‚7648Ý*<'×ÿM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚509235059443200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7384236922880 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125604924 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7700827493888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125604926 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 285066263 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501850804517376 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159461337 _runtime‚7648.901651575 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚501850822520320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7357650538602 trainer/global_step‚7199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚557 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111927115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39932307 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7700624976384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚489086118782432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7356879889982 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7383464264704 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159438295 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111927672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71995363 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 285043219 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159461337 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 285066263 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚489086118782432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7356879889982 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7383464264704 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159438295 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111927672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71995363 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 285043219 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159461337 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 285066263 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125604926 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 285066263 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501850804517376 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159461337 _runtime‚7648.901651575 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚501850822520320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7357650538602 trainer/global_step‚7199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚557 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111927115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39932307 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7700624976384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 P¸h2Ù- _wandbruntime‚7648  _runtime‚764894h\ÚÖ 1loss/FBt3/assignment_loss‚0.11914329952313096 1loss/SRqqt4/detection_loss‚0.7681098412722349 % _timestamp‚1.7884955492641485e+09 1loss/FBt2/assignment_loss‚0.21302272594746577 1loss/FBt4/assignment_loss‚0.11439773142954386 2loss/SRqqt1/assignment_loss‚1.6717876703650862 2loss/SRqqt2/assignment_loss‚1.9744688818852107 &loss/total_loss‚40.599503211167445 _runtime‚7648.904523056 /loss/FRt1/assignment_loss‚3.384212589263916 0loss/FRt4/assignment_loss‚0.8517148073377281 .loss/FBt2/detection_loss‚2.576815201685979 2loss/SRqqt4/assignment_loss‚0.8530787666607648 1loss/SRqqt2/detection_loss‚2.2676403265860348 .loss/FBt1/detection_loss‚2.324628233909607 0loss/FRt2/assignment_loss‚3.2061728835105896 1loss/SRqqt3/assignment_loss‚1.536559895888247 /loss/FRt4/detection_loss‚1.5651225319196438 0loss/SRqqt1/detection_loss‚2.601492245992025 epoch‚5 trainer/global_step‚7199 1loss/SRqqt3/detection_loss‚1.8337145276643612 .loss/FBt3/detection_loss‚2.464648043115934 _step‚590 /loss/FRt3/assignment_loss‚2.129793024376819 1loss/FBt1/assignment_loss‚0.40612574349580655 /loss/FRt1/detection_loss‚2.2545247736904357 /loss/FRt2/detection_loss‚2.6380487504148014 .loss/FRt3/detection_loss‚2.296538555896596 /loss/FBt4/detection_loss‚0.5477421593354785g—ðÇaÛÛ 1loss/FBt2/assignment_loss‚0.21302272594746577 1loss/FBt4/assignment_loss‚0.11439773142954386 2loss/SRqqt1/assignment_loss‚1.6717876703650862 2loss/SRqqt2/assignment_loss‚1.9744688818852107 &loss/total_loss‚40.599503211167445 _runtime‚7648.904523056 /loss/FRt1/assignment_loss‚3.384212589263916 0loss/FRt4/assignment_loss‚0.8517148073377281 .loss/FBt2/detection_loss‚2.576815201685979 2loss/SRqqt4/assignment_loss‚0.8530787666607648 1loss/SRqqt2/detection_loss‚2.2676403265860348 .loss/FBt1/detection_loss‚2.324628233909607 0loss/FRt2/assignment_loss‚3.2061728835105896 1loss/SRqqt3/assignment_loss‚1.536559895888247 /loss/FRt4/detection_loss‚1.5651225319196438 0loss/SRqqt1/detection_loss‚2.601492245992025 epoch‚5 trainer/global_step‚7199 1loss/SRqqt3/detection_loss‚1.8337145276643612 .loss/FBt3/detection_loss‚2.464648043115934 _step‚590 /loss/FRt3/assignment_loss‚2.129793024376819 1loss/FBt1/assignment_loss‚0.40612574349580655 /loss/FRt1/detection_loss‚2.2545247736904357 /loss/FRt2/detection_loss‚2.6380487504148014 .loss/FRt3/detection_loss‚2.296538555896596 /loss/FBt4/detection_loss‚0.5477421593354785 1loss/FBt3/assignment_loss‚0.11914329952313096 1loss/SRqqt4/detection_loss‚0.7681098412722349 % _timestamp‚1.7884955492641485e+09ζ2ˆÜ‚(:ý ¾éÔõÙ¸²' disk.nvme0n1.out‚184228.94775390625$ disk.dm-0.out‚184228.94775390625 proc.cpu.threads‚17 network.sent‚ 1.7814156e+07 disk.sda3.out‚ 548.68359375 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.0859375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚92.64671802520752 disk.sda3.in‚0 disk.dm-1.out‚ 548.68359375 disk.sda1.out‚0" disk.nvme0n1.in‚26391.69140625 disk.dm-0.in‚26391.69140625) proc.memory.availableMB‚ 1204.76171875) proc.memory.percent‚17.46535301208496 cpu‚25.264403695675995 disk.drbd1260.in‚0 disk.sda.out‚ 548.68359375$ disk.nvme0c0n1.in‚26391.69140625) disk.nvme0c0n1.out‚184228.94775390625" proc.memory.rssMB‚ 2861.5234375 network.recv‚ 2.565948e+06) disk./.usagePercent‚78.77492786322104$ disk./.usageGB‚1173.803352355957 disk.sda.in‚0—f릒Ý‚(:‡ ¾éÔ›¿â³ gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙÑނ(:† ÍéÔÓ·¯® gpu.0.gpu‚9.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.673' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.9346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01*á~‡ß‚(:ü ÍéÔ¿àŲ network.sent‚ 1.7923239e+07) disk.nvme0c0n1.out‚184462.32666015625 disk.dm-0.in‚26391.69140625 disk.drbd1260.in‚0( proc.memory.availableMB‚ 1204.5078125) proc.memory.percent‚17.46535301208496 proc.cpu.threads‚17( disk./.usagePercent‚78.7733477847197% disk./.usageGB‚1173.7798080444336 disk.dm-1.in‚0 disk.sda.out‚ 549.44140625 cpu‚25.264116487707863 network.recv‚ 2.572317e+06" disk.nvme0n1.in‚26391.69140625' disk.nvme0n1.out‚184462.32666015625$ disk.dm-0.out‚184462.32666015625 disk.sda3.in‚0 disk.dm-1.out‚ 549.44140625 disk.drbd1260.out‚ 1.0859375 disk.sda1.in‚0$ disk.nvme0c0n1.in‚26391.69140625 disk.sda3.out‚ 549.44140625 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚92.64826774597168" proc.memory.rssMB‚ 2861.5234375óÕ+’à‚(:‡ ÜéÔôÊÏ® gpu.0.gpu‚47.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.517' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5034 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NJj\‚á‚(:÷ ÜéÔ¿Ö»³ disk.dm-0.in‚ 26518.4609375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.7930921e+07# disk.nvme0c0n1.in‚ 26518.4609375( disk.nvme0c0n1.out‚184728.2744140625# disk.dm-0.out‚184728.2744140625 disk.drbd1260.out‚ 1.08984375 disk.sda.out‚ 550.03515625( proc.memory.availableMB‚ 1204.4921875" proc.memory.rssMB‚ 2861.5234375 network.recv‚ 2.576765e+06) disk./.usagePercent‚78.77492274307035 disk.sda3.out‚ 550.03515625 disk.drbd1260.in‚0) proc.memory.percent‚17.46535301208496% disk./.usageGB‚1173.8032760620117 disk.dm-1.out‚ 550.03515625 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚92.64836311340332 cpu‚25.26379295639637 proc.cpu.threads‚17! disk.nvme0n1.in‚ 26518.4609375& disk.nvme0n1.out‚184728.2744140625FjÆ’â‚(:‡ ëéÔé¢í­ gpu.0.gpu‚31.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.994' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?†à‚ã‚(:÷ ëéÔãú®& disk.nvme0n1.out‚184889.4541015625" proc.memory.rssMB‚ 2861.5234375 network.sent‚ 1.7934255e+07% disk./.usageGB‚1173.7798080444336 disk.dm-1.in‚0 disk.sda.out‚ 550.34765625 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 26518.4609375$ memory_percent‚92.64674186706543 network.recv‚ 2.580163e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.08984375 disk.sda1.out‚0( disk.nvme0c0n1.out‚184889.4541015625! disk.nvme0n1.in‚ 26518.4609375# disk.dm-0.out‚184889.4541015625 proc.cpu.threads‚17( disk./.usagePercent‚78.7733477847197 disk.sda3.out‚ 550.34765625 disk.dm-1.out‚ 550.34765625 disk.sda.in‚0 disk.dm-0.in‚ 26518.4609375( proc.memory.availableMB‚ 1204.7578125) proc.memory.percent‚17.46535301208496 cpu‚25.263475412761903 disk.sda3.in‚0 disk.sda2.out‚0Ô³¿•2ä- _wandbruntime‚7697  _runtime‚7697Ë%.Èå ' lr-AdamW/pg1‚0.00032789858467290414 ' lr-AdamW/pg2‚0.00032789858467290414 trainer/global_step‚7249 $ _timestamp‚1.788495598061192e+09 _runtime‚7697.701820976 _step‚591oU @ÍæÇ $ _timestamp‚1.788495598061192e+09 _runtime‚7697.701820976 _step‚591 ' lr-AdamW/pg1‚0.00032789858467290414 ' lr-AdamW/pg2‚0.00032789858467290414 trainer/global_step‚7249Ï Ä&ß2ç- _wandbruntime‚7697  _runtime‚7697‚‰¬ÿ\èôO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126431752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚568844393364992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72468117 =7Dev¨ÐgNž&iceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 # _timestamp‚1.78849559806275e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160528502 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512600812763648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚561092085662720 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7406719983624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 trainer/global_step‚7249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚505168101544448 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126431750 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72468115 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚505168119547392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286937236 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚499729323469884 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚512601603411456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7432711219200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚7697.703387317 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7405949335212 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7432711219200 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160505486 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112662506 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚568844611447296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7433483864064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846äâßYÿ'éùO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚561092109602816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40194389 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286960254 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40193840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 112661955 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126431752 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160528502 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7752307702272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126431752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚568844393364992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72468117 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 # _timestamp‚1.78849559806275e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160528502 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512600812763648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚561092085662720 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7406719983624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 trainer/global_step‚7249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚505168101544448 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126431750 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72468115 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚505168119547392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286937236 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚499729323469884 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚512601603411456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7432711219200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚7697.703387317 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7405949335212 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7432711219200 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160505486 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112662506 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚568844611447296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7433483864064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126448624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚556 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚561167457945600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚505235819486720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7433715964928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7406950475090 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚505235819486720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 trainer/global_step‚7249 _step‚593 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160550239 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚505235801483776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚499797178733502 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286998865 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160550239 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7753368050176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7433715964928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126448626 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112676876 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40199667 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚492389440570432 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126448626 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112676876 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40199667 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚492389440570432 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126448624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚556 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚561167457945600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚505235819486720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7433715964928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7406950475090 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚505235819486720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 trainer/global_step‚7249 _step‚593 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160550239 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚505235801483776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚499797178733502 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286998865 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160550239 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7753368050176Ñiœ W2í- _wandbruntime‚7698  _runtime‚7698²"úRZîÔ % _timestamp‚1.7884955990897322e+09 0loss/FRt1/assignment_loss‚3.7388597249984743 /loss/FRt4/assignment_loss‚1.103811874109156 2loss/SRqqt3/assignment_loss‚1.4108912251409023 .loss/FRt2/detection_loss‚2.386747427341971 1loss/SRqqt1/detection_loss‚2.3047701098300792 trainer/global_step‚7249 /loss/FRt4/detection_loss‚1.3912394944687976 1loss/SRqqt4/detection_loss‚0.8314776044379879 /loss/FRt3/detection_loss‚2.6297187554208854 1loss/SRqqt3/detection_loss‚1.6298030416571325 epoch‚5 0loss/FRt3/assignment_loss‚2.3056006070814634 0loss/FBt1/assignment_loss‚0.2432619949270572 /loss/FBt4/detection_loss‚0.7790577081802955 _runtime‚7698.730218642 .loss/FBt1/detection_loss‚2.510671649660383 _step‚594 2loss/SRqqt2/assignment_loss‚1.7323971680232457 1loss/FBt4/assignment_loss‚0.11589279950653485 .loss/FRt1/detection_loss‚2.479438157379627 0loss/SRqqt2/detection_loss‚2.466646468639374 .loss/FBt2/detection_loss‚3.685636651515961 1loss/SRqqt1/assignment_loss‚1.113125424625145 2loss/SRqqt4/assignment_loss‚1.0053941419573276 0loss/FBt2/assignment_loss‚0.3891328066121787 1loss/FBt3/assignment_loss‚0.07116859258343311 /loss/FBt3/detection_loss‚1.6477168872952461 %loss/total_loss‚41.48070035047863 0loss/FRt2/assignment_loss‚3.5082400350859673t=œ_ïÙ 0loss/FRt1/assignment_loss‚3.7388597249984743 /loss/FRt4/assignment_loss‚1.103811874109156 2loss/SRqqt3/assignment_loss‚1.4108912251409023 .loss/FRt2/detection_loss‚2.386747427341971 1loss/SRqqt1/detection_loss‚2.3047701098300792 trainer/global_step‚7249 /loss/FRt4/detection_loss‚1.3912394944687976 1loss/SRqqt4/detection_loss‚0.8314776044379879 /loss/FRt3/detection_loss‚2.6297187554208854 1loss/SRqqt3/detection_loss‚1.6298030416571325 epoch‚5 0loss/FRt3/assignment_loss‚2.3056006070814634 0loss/FBt1/assignment_loss‚0.2432619949270572 /loss/FBt4/detection_loss‚0.7790577081802955 _runtime‚7698.730218642 .loss/FBt1/detection_loss‚2.510671649660383 _step‚594 2loss/SRqqt2/assignment_loss‚1.7323971680232457 1loss/FBt4/assignment_loss‚0.11589279950653485 .loss/FRt1/detection_loss‚2.479438157379627 0loss/SRqqt2/detection_loss‚2.466646468639374 .loss/FBt2/detection_loss‚3.685636651515961 1loss/SRqqt1/assignment_loss‚1.113125424625145 2loss/SRqqt4/assignment_loss‚1.0053941419573276 0loss/FBt2/assignment_loss‚0.3891328066121787 1loss/FBt3/assignment_loss‚0.07116859258343311 /loss/FBt3/detection_loss‚1.6477168872952461 %loss/total_loss‚41.48070035047863 0loss/FRt2/assignment_loss‚3.5082400350859673 % _timestamp‚1.7884955990897322e+09Òq"ç-‘ð‚(:† úéÔ°ý³¯ gpu.0.gpu‚7.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.147' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0294 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©H“þ}ñ‚(:ò úéÔŸÜê° network.sent‚ 1.7952263e+07! disk.nvme0n1.in‚ 26518.4609375' disk.nvme0n1.out‚185243.51904296875 disk.dm-0.in‚ 26518.4609375 disk.drbd1260.out‚1.09375 disk.sda.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1125.1953125 network.recv‚ 2.583715e+06 disk.sda3.in‚0 disk.sda3.out‚ 551.015625 disk.dm-1.in‚0$ memory_percent‚93.13235282897949) proc.memory.percent‚17.94741153717041 cpu‚25.26744808921961 proc.cpu.threads‚17) disk./.usagePercent‚78.77492709519844# disk.nvme0c0n1.in‚ 26518.4609375) disk.nvme0c0n1.out‚185243.51904296875# proc.memory.rssMB‚ 2940.50390625% disk./.usageGB‚1173.8033409118652$ disk.dm-0.out‚185243.51904296875 disk.dm-1.out‚ 551.015625 disk.drbd1260.in‚0 disk.sda.out‚ 551.015625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0#-L0’ò‚(:‡ ‰ŽéÔÁŠ¾® gpu.0.gpu‚54.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚181.142' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.2284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘$¸‚ó‚(:÷ ‰ŽéÔüŠ¡± disk.dm-0.in‚ 26518.4609375$ disk.dm-0.out‚185355.68310546875 disk.dm-1.in‚0 disk.drbd1260.out‚1.09375 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.out‚ 551.81640625$ memory_percent‚93.18537712097168 network.sent‚ 1.805035e+07$ disk./.usageGB‚1173.555908203125! disk.nvme0n1.in‚ 26518.4609375' disk.nvme0n1.out‚185355.68310546875 disk.dm-1.out‚ 551.81640625 disk.sda1.in‚0# proc.memory.rssMB‚ 2949.55078125 cpu‚25.27743692817578) disk./.usagePercent‚78.75832167850501# disk.nvme0c0n1.in‚ 26518.4609375) disk.nvme0c0n1.out‚185355.68310546875 disk.drbd1260.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1116.5078125* proc.memory.percent‚18.002629280090332 proc.cpu.threads‚17 disk.sda3.out‚ 551.81640625 network.recv‚ 2.590785e+06 disk.sda2.in‚0"ïe‘ô‚(:† ˜ŽéÔÅîâ­ gpu.0.gpu‚54.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.401 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y[‚õ‚(:÷ ˜ŽéÔÑ©Ÿ² disk.sda.out‚ 552.21875$ memory_percent‚93.18687915802002 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 552.21875 disk.dm-1.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1116.26171875# proc.memory.rssMB‚ 2949.55078125 cpu‚25.285018591912152 network.sent‚ 1.8056256e+07$ disk./.usageGB‚1172.692512512207 disk.dm-0.in‚26670.16796875 disk.drbd1260.out‚ 1.09765625 disk.sda.in‚0 disk.sda1.in‚0 network.recv‚ 2.594965e+06) disk./.usagePercent‚78.70037846925025$ disk.nvme0c0n1.in‚26670.16796875" disk.nvme0n1.in‚26670.16796875' disk.nvme0n1.out‚185752.84619140625 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚18.002629280090332) disk.nvme0c0n1.out‚185752.84619140625$ disk.dm-0.out‚185752.84619140625 disk.dm-1.out‚ 552.21875 disk.drbd1260.in‚0!OÖ»’ö‚(:‡ §ŽéÔý»ç² gpu.0.gpu‚54.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.193' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=‹[Ú÷‚(:ö §ŽéÔïë±´ disk.drbd1260.out‚ 1.09765625 disk.sda.out‚ 552.90625 disk.sda1.out‚0" disk.nvme0n1.in‚26670.16796875 disk.dm-0.in‚26670.16796875$ memory_percent‚93.18690299987793 network.sent‚ 1.806146e+07 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 552.90625 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚26670.16796875$ disk.dm-0.out‚185985.15869140625# proc.memory.rssMB‚ 2949.55078125 network.recv‚ 2.598538e+06) disk./.usagePercent‚78.70188814567967% disk./.usageGB‚1172.7150077819824 disk.sda.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1116.2578125* proc.memory.percent‚18.002629280090332 cpu‚25.291382223722056 disk.sda1.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚185985.15869140625' disk.nvme0n1.out‚185985.15869140625 proc.cpu.threads‚17 disk.sda3.out‚ 552.90625S^2ø- _wandbruntime‚7760  _runtime‚7760Æq%b*ùÜT WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚515195246597120 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7433857148990 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 127151748 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18215 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚507734551206400 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚39 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚40472215 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578072504 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _step‚595 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚571551326529536 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 7434435221494 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 580044288 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 288175928 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚40472395 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7780696730112 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10558147584 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 7461293437952 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 161024180 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚72907377 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634825 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 598047232 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚507734551206400 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1005 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 150636032 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚515195246597120 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _runtime‚7760.318962576 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 161042395 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882825 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 113379590 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12155092992 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 $ _timestamp‚1.788495660678183e+09 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 288175928 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 127151750 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚494726202478176 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚515195844644352 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12155092992 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18217 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111864 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚502160059627166 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚494726219517536 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 598047232 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 7460713393664 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 126695936 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 161024180 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚563770780435456 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664Á›˜ÖÏúáT ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚515195246597120 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _runtime‚7760.318962576 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 161042395 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882825 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 113379590 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12155092992 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 $ _timestamp‚1.788495660678183e+09 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 288175928 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 127151750 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚494726202478176 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚515195844644352 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12155092992 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18217 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111864 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚502160059627166 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚494726219517536 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 598047232 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 7460713393664 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 126695936 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 161024180 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚563770780435456 [KDeviceStøá4˜atsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚515195246597120 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7433857148990 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 127151748 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18215 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚507734551206400 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚39 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚40472215 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578072504 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _step‚595 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚571551326529536 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 7434435221494 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 580044288 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 288175928 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚40472395 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7780696730112 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10558147584 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 7461293437952 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 161024180 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚72907377 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634825 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 598047232 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚507734551206400 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1005 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 150636032 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1019674624Óºfª2û- _wandbruntime‚7761  _runtime‚77617äìm)üçR M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 288197194 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18215 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595111864 % _timestamp‚1.7884956617810075e+09 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 127164720 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚563817734640640 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18217 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18215 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 7461758186496 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚571598920059392 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 7781161478656 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 161050689 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 288197194 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18217 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 580044288 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚516 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7461178142208 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1046 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3125 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚40477588 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 126695936 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 161032474 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚563817758580736 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1005 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28115 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 10929229312 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 161032474 _step‚596 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚183 _runtime‚7761.421776307 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580044288 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚39 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚502202818628226 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 598047232 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 113392762 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚40477407 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27846 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 161050689 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7781034782720 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12155092992 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1005 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 113392945 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 7434898382458 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 11892004352 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3125 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1005 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 127164720 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7461758186496 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1075 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚181 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚571598769423360 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12155092992 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚502203413740090 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 7434320309954 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 578072504 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚72915355 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 127164722 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚494768515357632 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 127164722 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27846 trainer/global_step‚7253 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚515241179344896 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 598047232 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚72915357 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 150636032 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚494768498318272 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚515241179344896 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚507780001202688‘kùîr)ýìR RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1046 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3125 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚40477588 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 126695936 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 161032474 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚563817758580736 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1005 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28115 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 10929229312 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 161032474 _step‚596 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚183 _runtime‚7761.421776307 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580044288 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚39 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚502202818628226 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 598047232 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 113392762 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚40477407 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27846 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 161050689 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7781034782720 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12155092992 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1005 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 113392945 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 7434898382458 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 11892004352 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3125 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1005 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 127164720 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7461758186496 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1075 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚181 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚571598769423360 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12155092992 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚502203413740090 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 7434320309954 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 578072504 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚72915355 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 127164722 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚494768515357632 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 127164722 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27846 trainer/global_step‚7253 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚515241179344896 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 598047232 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚72915357 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 150636032 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚494768498318272 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚515241179344896 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚507780001202688 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 288197194 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18215 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595111864 % _timestamp‚1.7884956617810075e+09 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 127164720 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚563817734640640 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18217 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18215 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 7461758186496 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚571598920059392 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 7781161478656 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 161050689 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 288197194 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18217 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 580044288 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚516 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7461178142208 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0ÔÈ’’þ‚(:‡ ¶ŽéÔòÊç¯ gpu.0.gpu‚52.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚184.721' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.9442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£y§á‡ÿ‚(:ü ¶ŽéÔÓíÜ´) disk./.usagePercent‚78.69091156664159% disk./.usageGB‚1172.5514488220215# proc.memory.rssMB‚ 2949.55078125 proc.cpu.threads‚17 network.sent‚ 1.8064514e+07) disk.nvme0c0n1.out‚186051.46337890625' disk.nvme0n1.out‚186051.46337890625$ disk.dm-0.out‚186051.46337890625 disk.sda3.in‚0 disk.dm-1.out‚ 553.3359375 disk.drbd1260.out‚ 1.1015625 disk.sda.in‚0 network.recv‚ 2.601925e+06$ disk.nvme0c0n1.in‚26670.16796875" disk.nvme0n1.in‚26670.16796875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚93.18859577178955 disk.dm-0.in‚26670.16796875 disk.sda3.out‚ 553.3359375 disk.sda.out‚ 553.3359375 disk.sda1.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1115.98046875* proc.memory.percent‚18.002629280090332 cpu‚25.29709676284416½AY2€ - _wandbruntime‚7776  _runtime‚7776Eê+'Ô á¥ ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007389721926301718 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5305381417274475 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12596702575683594 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.035412516444921494 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5545350909233093 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6159722208976746 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2617746591567993 +jet/accuracy_2_of_3‚0.26824164390563965 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_6_of_7‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 *jet/accuracy_1_of_4‚0.8962893486022949 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11131030321121216 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9907185435295105 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.35923007130622864 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5175642371177673 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3314994275569916 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.457023561000824 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2767857015132904 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2432747185230255 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.44430533051490784 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.03040560893714428 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8200281262397766 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.531827986240387 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3238278925418854 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_6‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.017473021522164345 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.26131686568260193 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_5‚1 'particle/f_score‚0.4062226712703705 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9885343313217163 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.21085982024669647 particle/accuracy_1_of_3‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.24000002443790436 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23004946112632751 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2559273838996887 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9881510734558105 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 ("Purity/3FRt1SRqqt1FBt/event_puK¾ }rity‚0 +jet/accuracy_3_of_6‚0.23076923191547394 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5085621476173401 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5592105388641357 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 jet/accuracy_4_of_7‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5309934020042419 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.22155171632766724 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.11087352782487869 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 jet/accuracy_1_of_7‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9841525554656982 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3032727837562561 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6052631735801697 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.992390513420105 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3480816185474396 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.42602360248565674 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9340277910232544 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.23755446076393127 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.472222238779068 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2652285397052765 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.48473167419433594 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.37011489272117615 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3857956528663635 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9196218252182007 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03230663016438484 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4392087459564209 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_1_of_3‚0.8594501614570618 jet/accuracy_3_of_7‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.38965076208114624 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05063505098223686 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4175436794757843 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 +particle/specificity‚0.9471001029014587 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17277030646800995 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3949335217475891 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9919275045394897 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.40918833017349243 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.318286657333374 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3627680540084839 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16893558204174042 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.22768794000148773 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5118975639343262 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_6‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.40822145342826843 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919540882110596 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4279699921607971 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.016807416453957558 particle/accuracy_1_of_2‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2941919267177582 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3542483150959015 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.237766295671463 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3947823643684387 *jet/accuracy_1_of_2‚0.7235603332519531 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2711547613143921 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2102481722831726 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3917911946773529 particle/accuracy_4_of_5‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011277956888079643 particle/accuracy_1_of_1‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.37092357873916626 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.36477214097976685 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5999999642372131 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_5_of_6‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 *jet/accuracy_1_of_1‚0.4526585638523102 *jet/accuracy_2_of_4‚0.3844633996486664 (particle/accuracy‚0.8504049777984619 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3759259283542633 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2183263599872589 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 _runtime‚ 7776.45204843 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.45956337451934814 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.25730058550834656 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4099215269088745 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.006896551698446274 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2541671693325043 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3534533679485321 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4291157126426697 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833432674408 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939395248889923 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41123509407043457 particle/accuracy_5_of_5‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5048742294311523 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020343536511063576 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37534716725349426 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7074470520019531 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3125 jet/accuracy_2_of_7‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3243224322795868 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_6‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5135658979415894 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 jet/accuracy_7_of_7‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9902756810188293 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4272462725639343 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.35168343782424927 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3591502606868744 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4876236617565155 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983958601951599 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.021447468549013138 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5473374724388123 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.09944581240415573 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9920634627342224 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.35517242550849915 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9934017658233643 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3976161479949951 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4029569923877716 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02026304230093956 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6044202446937561 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_2_of_6‚1 particle/accuracy_7_of_7‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4231877624988556 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3947089612483978 particle/accuracy_2_of_4‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 * Purity/3FRt1SRqqt1FBt/FRt_purity‚0.375 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12765957415103912 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ,particle/sensitivity‚0.32790249586105347 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07609239220619202 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26917970180511475 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23698298633098602 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974761605262756 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1794293373823166 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.20258881151676178 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2939189076423645 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2867525517940521 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5852470993995667 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9950113296508789 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3481399118900299 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3917911946773529 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.960983157157898 7validation_average_jet_accuracy‚0.41978779435157776 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28888893127441406 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 _step‚597 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0477818101644516 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.051735278218984604 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.291340708732605 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6443735361099243 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5863127112388611 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.024570126086473465 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3263888955116272 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.41807782649993896 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3776785731315613 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4211658835411072 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.25794994831085205 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 particle/accuracy_3_of_7‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.25484225153923035 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333134651184 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4117225408554077 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333134651184 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24239839613437653 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1471896916627884 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.986527144908905 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9378306865692139 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04148228093981743 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 particle/accuracy_1_of_7‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2075783908367157 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20872440934181213 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09646262973546982 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 jet/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019841270986944437 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9931153655052185 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.43852487206459045 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08883928507566452 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7196969389915466 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.40810462832450867 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5233333110809326 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968628287315369 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.42629069089889526 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.37151679396629333 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5141210556030273 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.37654322385787964 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5681561827659607 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 particle/accuracy_5_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5861303806304932 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29743245244026184 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23938456177711487 +jet/accuracy_2_of_5‚0.48865213990211487 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4315057694911957 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1807367354631424 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2843873202800751 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2470436692237854 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.27203065156936646 particle/accuracy_2_of_3‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2839505970478058 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4663703143596649 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9903976321220398 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36018985509872437 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6186354756355286 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.40123456716537476 +jet/accuracy_3_of_5‚0.11167433857917786 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4454430639743805 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 % _timestamp‚1.7884956768098078e+09 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9005847573280334 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47729891538619995 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20350876450538635 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 trainer/global_step‚7253 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.49537038803100586 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20650513470172882 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5470742583274841 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4142603576183319 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.43390804529190063 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 )loss/val_total_loss‚40.35122893674126 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3863636255264282 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10958102345466614 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.46143442392349243 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9891974925994873 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.21402302384376526 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3128902316093445 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_4_of_5‚0.014132104814052582 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3216843605041504 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6269841194152832 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.6410256028175354 jet/accuracy_4_of_6‚0 particle/accuracy_6_of_6‚1 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 epoch‚5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877755641937256 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 jet/accuracy_5_of_5‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4011527895927429 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.049046341329813004 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27023056149482727 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.052515722811222076 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24239839613437653 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0Á…Þ‚ æ¥ 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2652285397052765 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.48473167419433594 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.37011489272117615 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3857956528663635 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 6Eù7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9196218252182007 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03230663016438484 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4392087459564209 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_1_of_3‚0.8594501614570618 jet/accuracy_3_of_7‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.38965076208114624 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05063505098223686 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4175436794757843 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 +particle/specificity‚0.9471001029014587 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17277030646800995 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3949335217475891 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9919275045394897 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.40918833017349243 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.318286657333374 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3627680540084839 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16893558204174042 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.22768794000148773 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5118975639343262 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_6‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.40822145342826843 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919540882110596 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4279699921607971 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.016807416453957558 particle/accuracy_1_of_2‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2941919267177582 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3542483150959015 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.237766295671463 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3947823643684387 *jet/accuracy_1_of_2‚0.7235603332519531 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2711547613143921 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2102481722831726 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3917911946773529 particle/accuracy_4_of_5‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011277956888079643 particle/accuracy_1_of_1‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.37092357873916626 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.36477214097976685 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5999999642372131 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_5_of_6‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 *jet/accuracy_1_of_1‚0.4526585638523102 *jet/accuracy_2_of_4‚0.3844633996486664 (particle/accuracy‚0.8504049777984619 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3759259283542633 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2183263599872589 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 _runtime‚ 7776.45204843 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.45956337451934814 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.25730058550834656 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4099215269088745 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.006896551698446274 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2541671693325043 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3534533679485321 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4291157126426697 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833432674408 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939395248889923 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41123509407043457 particle/accuracy_5_of_5‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5048742294311523 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020343536511063576 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.37534716725349426 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7074470520019531 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3125 jet/accuracy_2_of_7‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3243224322795868 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_6‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5135658979415894 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 jet/accuracy_7_of_7‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9902756810188293 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4272462725639343 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.35168343782424927 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3591502606868744 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4876236617565155 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983958601951599 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.021447468549013138 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5473374724388123 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.09944581240415573 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9920634627342224 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.35517242550849915 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9934017658233643 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3976161479949951 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4029569923877716 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02026304230093956 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6044202446937561 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_2_of_6‚1 particle/accuracy_7_of_7‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4231877624988556 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3947089612483978 particle/accuracy_2_of_4‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 * Purity/3FRt1SRqqt1FBt/FRt_purity‚0.375 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12765957415103912 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ,particle/sensitivity‚0.32790249586105347 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07609239220619202 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26917970180511475 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23698298633098602 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974761605262756 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1794293373823166 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.20258881151676178 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2939189076423645 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2867525517940521 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5852470993995667 +"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9950113296508789 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3481399118900299 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3917911946773529 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.960983157157898 7validation_average_jet_accuracy‚0.41978779435157776 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28888893127441406 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 _step‚597 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0477818101644516 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.051735278218984604 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.291340708732605 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6443735361099243 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5863127112388611 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.024570126086473465 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3263888955116272 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.41807782649993896 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3776785731315613 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4211658835411072 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.25794994831085205 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 particle/accuracy_3_of_7‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.25484225153923035 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333134651184 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4117225408554077 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333134651184 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24239839613437653 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1471896916627884 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.986527144908905 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9378306865692139 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04148228093981743 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 particle/accuracy_1_of_7‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2075783908367157 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20872440934181213 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09646262973546982 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 jet/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019841270986944437 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9931153655052185 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.43852487206459045 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08883928507566452 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7196969389915466 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.40810462832450867 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5233333110809326 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968628287315369 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.42629069089889526 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.37151679396629333 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5141210556030273 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.37654322385787964 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5681561827659607 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 particle/accuracy_5_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5861303806304932 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29743245244026184 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23938456177711487 +jet/accuracy_2_of_5‚0.48865213990211487 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4315057694911957 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1807367354631424 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2843873202800751 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2470436692237854 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.27203065156936646 particle/accuracy_2_of_3‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2839505970478058 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4663703143596649 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9903976321220398 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36018985509872437 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6186354756355286 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.40123456716537476 +jet/accuracy_3_of_5‚0.11167433857917786 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4454430639743805 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 % _timestamp‚1.7884956768098078e+09 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9005847573280334 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47729891538619995 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20350876450538635 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 trainer/global_step‚7253 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.49537038803100586 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20650513470172882 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5470742583274841 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4142603576183319 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.43390804529190063 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 )loss/val_total_loss‚40.35122893674126 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3863636255264282 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10958102345466614 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.46143442392349243 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9891974925994873 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.21402302384376526 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3128902316093445 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_4_of_5‚0.014132104814052582 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3216843605041504 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6269841194152832 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.6410256028175354 jet/accuracy_4_of_6‚0 particle/accuracy_6_of_6‚1 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 epoch‚5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877755641937256 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 jet/accuracy_5_of_5‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4011527895927429 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.049046341329813004 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27023056149482727 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.052515722811222076 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24239839613437653 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007389721926301718 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5305381417274475 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12596702575683594 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.035412516444921494 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5545350909233093 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6159722208976746 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2617746591567993 +jet/accuracy_2_of_3‚0.26824164390563965 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_6_of_7‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 *jet/accuracy_1_of_4‚0.8962893486022949 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11131030321121216 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9907185435295105 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.35923007130622864 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5175642371177673 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3314994275569916 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.457023561000824 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2767857015132904 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2432747185230255 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.44430533051490784 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.03040560893714428 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8200281262397766 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.531827986240387 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3238278925418854 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuraĹŒ³cy_3_of_3‚1 particle/accuracy_3_of_6‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.017473021522164345 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.26131686568260193 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_5‚1 'particle/f_score‚0.4062226712703705 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9885343313217163 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.21085982024669647 particle/accuracy_1_of_3‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.24000002443790436 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23004946112632751 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2559273838996887 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9881510734558105 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.23076923191547394 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5085621476173401 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5592105388641357 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 jet/accuracy_4_of_7‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5309934020042419 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.22155171632766724 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.11087352782487869 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 jet/accuracy_1_of_7‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9841525554656982 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3032727837562561 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6052631735801697 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.992390513420105 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3480816185474396 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.42602360248565674 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9340277910232544 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.23755446076393127 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.472222238779068 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824Õ* Mh‰ƒ ‚(:þ ÅŽéÔôÊÞ°# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613) disk./.usagePercent‚78.69832016867497 disk.sda3.out‚ 553.81640625 disk.drbd1260.out‚ 1.1015625 disk.sda.out‚ 553.81640625 proc.cpu.threads‚17) disk.nvme0c0n1.out‚186975.56494140625' disk.nvme0n1.out‚186975.56494140625 disk.drbd1260.in‚0 disk.sda2.in‚0# memory_percent‚89.7470235824585) proc.memory.availableMB‚ 1679.84765625 cpu‚25.29913132169979 network.sent‚ 1.8173343e+07 network.recv‚ 2.608649e+06% disk./.usageGB‚1172.6618423461914$ disk.nvme0c0n1.in‚26670.16796875$ disk.dm-0.out‚186975.56494140625 disk.sda3.in‚0 disk.dm-1.out‚ 553.81640625 disk.sda1.in‚0" disk.nvme0n1.in‚26670.16796875 disk.dm-0.in‚26670.16796875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0§Ý†N„ ‚(:’ ÅŽéÔøŽØ° gpu.0.gpu‚24.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.069' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.213800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚¸*’… ‚(:‡ ÔŽéÔ½†ì­ gpu.0.gpu‚23.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.916' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9832 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç¨ ô†† ‚(:û ÔŽéÔ€ÍØ° network.sent‚ 1.8297253e+07) disk./.usagePercent‚78.70877679641102% disk./.usageGB‚1172.8176536560059 disk.sda2.out‚0' disk.nvme0n1.out‚187024.39306640625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.10546875 network.recv‚ 2.61483e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 554.3046875 disk.sda.in‚0 disk.sda.out‚ 554.3046875( proc.memory.availableMB‚ 1679.2890625# proc.memory.rssMB‚ 2949.58203125$ disk.nvme0c0n1.in‚26808.94140625 disk.dm-0.in‚26808.94140625 disk.sda3.out‚ 554.3046875 disk.drbd1260.in‚0* proc.memory.percent‚18.002820014953613 cpu‚25.29875269381812 disk.sda2.in‚0) disk.nvme0c0n1.out‚187024.39306640625" disk.nvme0n1.in‚26808.94140625$ disk.dm-0.out‚187024.39306640625$ memory_percent‚89.75043296813965 proc.cpu.threads‚17’±‡ ‚(:’ ãŽéÔÌ—Ì­ gpu.0.gpu‚43.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.427' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.885399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fŸf*ˆˆ ‚(:ý ãŽéÔ¦·ë°) disk.nvme0c0n1.out‚187804.40478515625# memory_percent‚89.7493839263916 network.sent‚ 1.830238e+07% disk./.usageGB‚1172.6974639892578 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚18.002820014953613( disk./.usagePercent‚78.7007107670297 disk.sda3.out‚ 554.66015625 disk.sda.out‚ 554.66015625 disk.sda1.in‚0$ disk.nvme0c0n1.in‚26808.94140625' disk.nvme0n1.out‚187804.40478515625 cpu‚25.298374080178824 network.recv‚ 2.618533e+06 disk.dm-0.in‚26808.94140625$ disk.dm-0.out‚187804.40478515625 disk.dm-1.out‚ 554.66015625" disk.nvme0n1.in‚26808.94140625( proc.memory.availableMB‚ 1679.4609375# proc.memory.rssMB‚ 2949.58203125 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.10546875 disk.sda2.out‚0Ý3‹f2‰ - _wandbruntime‚7825  _runtime‚7825H5ÇŠ Á trainer/global_step‚7299 % _timestamp‚1.7884957256854627e+09 _runtime‚7825.325906357 _step‚598 & lr-AdamW/pg1‚0.0003222728789022833 & lr-AdamW/pg2‚0.0003222728789022833r—º%Ì‹ Æ _step‚598 & lr-AdamW/pg1‚0.0003222728789022833 & lr-AdamW/pg2‚0.0003222728789022833 trainer/global_step‚7299 % _timestamp‚1.7884957256854627e+09 _runtime‚7825.325906357ÖAª ¹2Œ - _wandbruntime‚7825  _runtime‚7825¶ÖÚ¹ü' öO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7513031713280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚567819149894656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚505777337704922 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚575653543966720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73453257 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚567819125954560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498291333848720 % _timestamp‚1.7884957256868644e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128092686 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162118847 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162118847 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚505776550017150 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚518930632178688 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128092686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7485986816842 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚511417600465408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7513031713280 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114232492 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7834394072064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚7825.327429297 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 _step‚599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290211533 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 290234551 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128092688 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚518929841530880 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162141863 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7834195735552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125LJl„(Ž ûO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚518930632178688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7512259068416 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114231968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚511417600465408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40779235 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73453255 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚518929841530880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40778713 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7485216168430 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚518930632178688 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128092686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7485986816842 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚511417600465408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7513031713280 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114232492 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7834394072064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚7825.327429297 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 _step‚599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290211533 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 290234551 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128092688 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚518929841530880 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162141863 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7834195735552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7513031713280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚567819149894656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚505777337704922 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚575653543966720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73453257 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚567819125954560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498291333848720 % _timestamp‚1.7884957256868644e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128092686 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162118847 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162118847 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚505776550017150 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚511485300404736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114247334 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _runtime‚7826.306913124 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290250118 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40784429 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128109562 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114246809 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚600 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚511485300404736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚505843617592788 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 % _timestamp‚1.7884957266654005e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498357417323840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚511485282401792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚518999336877056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ ¿æ€128109560 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290273162 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162163600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40783906 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7513263814144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚575729731872768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73462905 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚511485282401792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290273162 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162140558 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73462903 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7835449607168 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498357400284480 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚567894498237440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7486217308308 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚567894474297344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162140558 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7514036472320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7835257575424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 trainer/global_step‚7299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0°óRÕ '‘ …N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290273162 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162140558 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73462903 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7835449607168 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498357400284480 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚567894498237440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7486217308308 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚567894474297344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162140558 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7514036472320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7835257575424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 trainer/global_step‚7299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚511485300404736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114247334 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _runtime‚7826.306913124 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290250118 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40784429 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128109562 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114246809 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚600 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚511485300404736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚505843617592788 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 % _timestamp‚1.7884957266654005e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498357417323840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚511485282401792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚518999336877056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128109560 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290273162 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162163600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40783906 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7513263814144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚575729731872768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73462905 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚511485282401792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664غéê#2’ - _wandbruntime‚7826  _runtime‚7826Ü°»`]“ × 2loss/SRqqt1/assignment_loss‚1.3315761054831523 0loss/FBt1/assignment_loss‚0.3007166409451101 0loss/FRt2/assignment_loss‚2.8099907021368704 1loss/FBt2/assignment_loss‚0.26367636067854505 /loss/FRt2/detection_loss‚2.4632911724508113 0loss/FRt4/assignment_loss‚1.0444070868767226 2loss/SRqqt4/assignment_loss‚0.8488514361431799 /loss/FRt1/detection_loss‚2.4688688268264136 .loss/FRt3/detection_loss‚2.603312118399528 1loss/SRqqt1/detection_loss‚2.5276579591963024 /loss/FBt3/detection_loss‚1.8350501446162952 /loss/FRt3/assignment_loss‚2.775532526354636 %loss/total_loss‚40.17243027628278 0loss/FRt1/assignment_loss‚3.2691123546698155 2loss/SRqqt3/assignment_loss‚1.5358324319405376 0loss/SRqqt2/detection_loss‚2.002513048740534 _runtime‚7826.308230426 1loss/FBt3/assignment_loss‚0.14102656029921062 /loss/FBt4/detection_loss‚0.6457510873608824 epoch‚6 $ _timestamp‚1.788495726667958e+09 1loss/FBt4/assignment_loss‚0.10299192313982268 1loss/SRqqt3/detection_loss‚1.6897613252952415 1loss/SRqqt4/detection_loss‚0.7104558102247944 .loss/FBt1/detection_loss‚2.303582297431098 /loss/FBt2/detection_loss‚3.3346690014004707 _step‚601 2loss/SRqqt2/assignment_loss‚1.5458605756314518 /loss/FRt4/detection_loss‚1.6179427800413508 trainer/global_step‚7299¡(ãb” Ü 0loss/FRt1/assignment_loss‚3.2691123546698155 2loss/SRqqt3/assignment_loss‚1.5358324319405376 0loss/SRqqt2/detection_loss‚2.002513048740534 _runtime‚7826.308230426 1loss/FBt3/assignment_loss‚0.14102656029921062 /loss/FBt4/detection_loss‚0.6457510873608824 epoch‚6 $ _timestamp‚1.788495726667958e+09 1loss/FBt4/assignment_loss‚0.10299192313982268 1loss/SRqqt3/detection_loss‚1.6897613252952415 1loss/SRqqt4/detection_loss‚0.7104558102247944 .loss/FBt1/detection_loss‚2.303582297431098 /loss/FBt2/detection_loss‚3.3346690014004707 _step‚601 2loss/SRqqt2/assignment_loss‚1.5458605756314518 /loss/FRt4/detection_loss‚1.6179427800413508 trainer/global_step‚7299 2loss/SRqqt1/assignment_loss‚1.3315761054831523 0loss/FBt1/assignment_loss‚0.3007166409451101 0loss/FRt2/assignment_loss‚2.8099907021368704 1loss/FBt2/assignment_loss‚0.26367636067854505 /loss/FRt2/detection_loss‚2.4632911724508113 0loss/FRt4/assignment_loss‚1.0444070868767226 2loss/SRqqt4/assignment_loss‚0.8488514361431799 /loss/FRt1/detection_loss‚2.4688688268264136 .loss/FRt3/detection_loss‚2.603312118399528 1loss/SRqqt1/detection_loss‚2.5276579591963024 /loss/FBt3/detection_loss‚1.8350501446162952 /loss/FRt3/assignment_loss‚2.775532526354636 %loss/total_loss‚40.17243027628278ÙÆÄÁ’• ‚(:‡ òŽéÔúþå® gpu.0.gpu‚14.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ë– ‚(:ö òŽéÔç disk.dm-1.out‚ 555.04296875 disk.drbd1260.in‚0" disk.nvme0n1.in‚26808.94140625# memory_percent‚89.7496223449707 disk.sda3.out‚ 555.04296875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚26808.94140625' disk.nvme0c0n1.out‚188086.529296875% disk.nvme0n1.out‚188086.529296875' proc.memory.availableMB‚ 1679.421875 network.recv‚ 2.622011e+06) disk./.usagePercent‚78.70858555878294% disk./.usageGB‚1172.8148040771484 disk.sda1.out‚0" disk.dm-0.out‚188086.529296875# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613 cpu‚25.297964825847473 network.sent‚ 1.8307421e+07 disk.drbd1260.out‚1.109375 disk.sda.in‚0 disk.sda.out‚ 555.04296875 disk.sda2.out‚0 disk.dm-0.in‚26808.94140625 proc.cpu.threads‚17 disk.sda3.in‚0~9œÅ— ‚(:’ éÔÒ÷ì¯ gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.123' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.224599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ˆäæ˜ ‚(:ö éÔ‚±é± cpu‚25.29764835686059 proc.cpu.threads‚17) disk./.usagePercent‚78.71015795705823 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚26808.94140625" disk.dm-0.out‚188294.771484375 disk.sda3.out‚ 555.69140625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 555.69140625 disk.drbd1260.out‚1.109375 disk.sda.out‚ 555.69140625 disk.sda1.out‚0$ memory_percent‚89.75071907043457# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613$ disk./.usageGB‚1172.838233947754' disk.nvme0c0n1.out‚188294.771484375% disk.nvme0n1.out‚188294.771484375 disk.dm-1.in‚0 network.sent‚ 1.8416317e+07 network.recv‚ 2.628677e+06$ disk.nvme0c0n1.in‚26808.94140625 disk.dm-0.in‚26808.94140625 disk.sda1.in‚0( proc.memory.availableMB‚ 1679.2421875|£óö’™ ‚(:‡ éÔâ È­ gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.661' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3322 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~!m€š ‚(:õ éÔ€ö° network.recv‚ 2.632821e+06% disk./.usageGB‚1172.6565895080566 disk.dm-1.in‚0 disk.dm-1.out‚ 556.48828125 disk.drbd1260.out‚ 1.11328125# proc.memory.rssMB‚ 2949.58203125 cpu‚25.297312201566523 proc.cpu.threads‚17 network.sent‚ 1.8424107e+07) disk./.usagePercent‚78.69796764630031 disk.sda3.in‚0 disk.sda3.out‚ 556.48828125 disk.nvme0c0n1.in‚ 26917.4375 disk.nvme0n1.in‚ 26917.4375$ memory_percent‚89.75088596343994) proc.memory.availableMB‚ 1679.21484375 disk.sda.out‚ 556.48828125 disk.dm-0.in‚ 26917.4375 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚188729.40087890625' disk.nvme0n1.out‚188729.40087890625* proc.memory.percent‚18.002820014953613$ disk.dm-0.out‚188729.40087890625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0"#eÌ’› ‚(:‡ ŸéÔ×ئ° gpu.0.gpu‚55.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.256' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÓÔA}œ ‚(:ò ŸéÔÖ‘Œ³ disk.sda1.out‚0 network.sent‚ 1.8427293e+07) disk./.usagePercent‚78.69943226540371% disk./.usageGB‚1172.6784133911133 disk.sda3.out‚ 556.765625 disk.dm-1.in‚0 disk.dm-1.out‚ 556.765625 disk.sda.out‚ 556.765625 network.recv‚ 2.636439e+06 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 26950.546875) disk.nvme0c0n1.out‚188729.74462890625 disk.nvme0n1.in‚ 26950.546875' disk.nvme0n1.out‚188729.74462890625 disk.sda3.in‚0 disk.dm-0.in‚ 26950.546875$ memory_percent‚89.75238800048828& proc.memory.availableMB‚ 1678.96875# proc.memory.rssMB‚ 2949.58203125 cpu‚25.296894416038356 proc.cpu.threads‚17 disk.drbd1260.out‚ 1.11328125 disk.sda1.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚188729.74462890625* proc.memory.percent‚18.002820014953613ï=Sz2 - _wandbruntime‚7877  _runtime‚7877:éV Çž Á & lr-AdamW/pg2‚0.0003166625629806927 trainer/global_step‚7349 % _timestamp‚1.7884957780525465e+09 _runtime‚7877.692951409 _step‚602 & lr-AdamW/pg1‚0.0003166625629806927µW„9ÌŸ Æ _step‚602 & lr-AdamW/pg1‚0.0003166625629806927 & lr-AdamW/pg2‚0.0003166625629806927 trainer/global_step‚7349 % _timestamp‚1.7884957780525465e+09 _runtime‚7877.692951409Ú««®v2  - _wandbruntime‚7877  _runtime‚7877šíW¢/¡ õO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7886490594304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚514802579428864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7563283413504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163230765 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163230765 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114976837 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚571586567033856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _step‚603 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114976298 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚579472841643008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292144135 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7563283413504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚579473057628160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7562510768640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73935655 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163207749 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514802579428864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7562510768640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128936386 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚509129942390258 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292167153 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522365090197504 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚522365090197504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41041180 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7886298549248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚509130730078030 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7536057401950 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163207749 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128936386 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514802597431808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;Devx©jÝÌ iceStatsMonitor.on_train_batch_start/segment.all.current‚1005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73935657 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚571586543093760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚7877.694134648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚522365880845312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚522365880845312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128936388 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514802597431808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 111295856647Ç[h(¢ úO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7562510768640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73935655 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163207749 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514802579428864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7562510768640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128936386 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚509129942390258 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292167153 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522365090197504 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚522365090197504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41041180 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7886298549248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚509130730078030 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7536057401950 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163207749 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128936386 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514802597431808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73935657 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚571586543093760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚7877.694134648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚522365880845312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚522365880845312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128936388 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514802597431808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7886490594304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚514802579428864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7563283413504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163230765 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163230765 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114976837 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚571586567033856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _step‚603 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114976298 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚579472841643008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292144135 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7563283413504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚579473057628160Ûo±Ð2£ - _wandbruntime‚7878  _runtime‚7878·òÞå'¤ ýM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128953260 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7564288172544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚501660722072480 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292205764 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522434585543680 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚579549453572096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 215100416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚604 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128953262 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 trainer/global_step‚7349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292182720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚522433794882560 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128953262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7563515514368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚509197009965896 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292205764 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7887323095040 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7563515514368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7536287893416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163229460 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114991634 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114991092 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7564288172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 239040512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73945303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571661915376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73945305 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 " _timestamp‚1.7884957788977e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292182720 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 _runtime‚7878.538763481Â1¦6Ü&¥ ‚N \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7564288172544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 239040512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73945303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571661915376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73945305 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 " _timestamp‚1.7884957788977e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292182720 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 _runtime‚7878.538763481 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128953260 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7564288172544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚501660722072480 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292205764 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522434585543680 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚579549453572096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 215100416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚604 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128953262 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 trainer/global_step‚7349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292182720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚522433794882560 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128953262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7563515514368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚509197009965896 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292205764 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7887323095040 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7563515514368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7536287893416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163229460 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114991634 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114991092 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 D>DeviceStatsMonitor.on_train×#vU,_batch_end/segment.large_pool.freed‚0Üó(¼2¦ - _wandbruntime‚7878  _runtime‚7878åbC_§ Ù 0loss/FBt1/assignment_loss‚0.2207926524179129 1loss/FBt3/assignment_loss‚0.12371885488025924 %loss/total_loss‚40.75191967885774 _step‚605 2loss/SRqqt4/assignment_loss‚0.7330493780389633 /loss/FRt1/detection_loss‚2.3205544281336996 /loss/FRt3/detection_loss‚2.3752275305715473 _runtime‚7878.540358325 1loss/SRqqt2/detection_loss‚2.1842405068523743 1loss/SRqqt3/detection_loss‚1.5719587353326507 /loss/FBt3/detection_loss‚2.6545675694942474 epoch‚6 1loss/FBt4/assignment_loss‚0.09263990998716722 /loss/FBt2/detection_loss‚2.6911976391618904 0loss/FRt3/assignment_loss‚2.5686336200345647 1loss/SRqqt1/assignment_loss‚1.168786737943689 /loss/FRt4/detection_loss‚1.7692499674777402 1loss/SRqqt4/detection_loss‚0.9261901719813667 $ _timestamp‚1.788495778900171e+09 0loss/FRt1/assignment_loss‚3.2353729128837587 2loss/SRqqt2/assignment_loss‚1.8955318162984707 2loss/SRqqt3/assignment_loss‚1.3556356483091743 .loss/FBt1/detection_loss‚2.735217014948527 0loss/FRt2/assignment_loss‚2.8572497690046155 0loss/FRt4/assignment_loss‚1.4014134051951956 1loss/FBt2/assignment_loss‚0.08519994358487731 /loss/FRt2/detection_loss‚2.5105300206191092 1loss/SRqqt1/detection_loss‚2.6471692860126494 /loss/FBt4/detection_loss‚0.6277921596932926 trainer/global_step‚7349ÁÀ™zd¨ Þ 0loss/FRt1/assignment_loss‚3.2353729128837587 2loss/SRqqt2/assignment_loss‚1.8955318162984707 2loss/SRqqt3/assignment_loss‚1.3556356483091743 .loss/FBt1/detection_loss‚2.735217014948527 0loss/FRt2/assignment_loss‚2.8572497690046155 0loss/FRt4/assignment_loss‚1.4014134051951956 1loss/FBt2/assignment_loss‚0.08519994358487731 /loss/FRt2/detection_loss‚2.5105300206191092 1loss/SRqqt1/detection_loss‚2.6471692860126494 /loss/FBt4/detection_loss‚0.6277921596932926 trainer/global_step‚7349 0loss/FBt1/assignment_loss‚0.2207926524179129 1loss/FBt3/assignment_loss‚0.12371885488025924 %loss/total_loss‚40.75191967885774 _step‚605 2loss/SRqqt4/assignment_loss‚0.7330493780389633 /loss/FRt1/detection_loss‚2.3205544281336996 /loss/FRt3/detection_loss‚2.3752275305715473 _runtime‚7878.540358325 1loss/SRqqt2/detection_loss‚2.1842405068523743 1loss/SRqqt3/detection_loss‚1.5719587353326507 /loss/FBt3/detection_loss‚2.6545675694942474 epoch‚6 1loss/FBt4/assignment_loss‚0.09263990998716722 /loss/FBt2/detection_loss‚2.6911976391618904 0loss/FRt3/assignment_loss‚2.5686336200345647 1loss/SRqqt1/assignment_loss‚1.168786737943689 /loss/FRt4/detection_loss‚1.7692499674777402 1loss/SRqqt4/detection_loss‚0.9261901719813667 $ _timestamp‚1.788495778900171e+09Ý’Œ”¹© ‚(:’ ®éÔÒËݯ gpu.0.gpu‚40.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.384' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.876799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•‰Í«€ª ‚(:õ ®éԵ啲# proc.memory.rssMB‚ 2949.58203125 cpu‚25.29649862088213 network.recv‚ 2.640047e+06% disk./.usageGB‚1172.9281692504883 disk.dm-1.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 26950.546875' disk.nvme0n1.out‚188986.55615234375 disk.sda3.in‚0 disk.sda3.out‚ 557.1328125 network.sent‚ 1.8432476e+07) disk./.usagePercent‚78.71619359068549 disk.dm-1.out‚ 557.1328125 disk.drbd1260.in‚0 disk.sda.out‚ 557.1328125 disk.sda1.out‚0 disk.nvme0n1.in‚ 26950.546875# memory_percent‚89.7512674331665 disk.drbd1260.out‚ 1.1171875 disk.sda2.out‚0 disk.dm-0.in‚ 26950.546875$ disk.dm-0.out‚188986.55615234375) proc.memory.availableMB‚ 1679.15234375* proc.memory.percent‚18.002820014953613 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚188986.55615234375|äky’« ‚(:‡ ½éÔˆ¹±­ gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚130.243' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û±[¬ ‚(:ö ½éÔ¸µ¨¸ disk.dm-0.in‚ 26950.546875$ memory_percent‚89.75281715393066# proc.memory.rssMB‚ 2949.58203125) disk./.usagePercent‚78.72454686052022% disk./.usageGB‚1173.0526390075684 disk.sda1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 1678.8984375 proc.cpu.threads‚17 disk.sda3.out‚ 557.9296875 disk.sda.in‚0 disk.sda.out‚ 557.9296875) disk.nvme0c0n1.out‚189272.43505859375' disk.nvme0n1.out‚189272.43505859375$ disk.dm-0.out‚189272.43505859375* proc.memory.percent‚18.002820014953613 cpu‚25.296121483404026 network.sent‚ 1.8541311e+07 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 26950.546875 disk.nvme0n1.in‚ 26950.546875 network.recv‚ 2.646837e+06 disk.sda3.in‚0 disk.dm-1.out‚ 557.9296875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.1171875$Ü£Ù‘­ ‚(:† ÌéÔûäÝ­ gpu.0.gpu‚41.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.935' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.187 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bìõ!|® ‚(:ñ ÌéÔêï¹± disk.sda.in‚0( proc.memory.availableMB‚ 1678.6484375# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613) disk./.usagePercent‚78.74966555175726 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 26950.546875$ disk.dm-0.out‚189678.90087890625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 26950.546875' disk.nvme0n1.out‚189678.90087890625 disk.sda3.out‚ 558.3359375 disk.dm-1.out‚ 558.3359375 disk.sda.out‚ 558.3359375$ memory_percent‚89.75434303283691 network.sent‚ 1.8547183e+07 network.recv‚ 2.651186e+06) disk.nvme0c0n1.out‚189678.90087890625 disk.dm-0.in‚ 26950.546875 disk.drbd1260.out‚1.125 cpu‚25.29576660586235 proc.cpu.threads‚17% disk./.usageGB‚1173.4269256591797¹ÝÕ2¯ - _wandbruntime‚7929  _runtime‚7929g¤ËÇ° Á % _timestamp‚1.7884958301740274e+09 _runtime‚7929.814373039 _step‚606 & lr-AdamW/pg1‚0.0003110688060895617 & lr-AdamW/pg2‚0.0003110688060895617 trainer/global_step‚7399Óðª•Ì± Æ _step‚606 & lr-AdamW/pg1‚0.0003110688060895617 & lr-AdamW/pg2‚0.0003110688060895617 trainer/global_step‚7399 % _timestamp‚1.7884958301740274e+09 _runtime‚7929.814373039ÞF4.32² - _wandbruntime‚7929  _runtime‚7929P} Aú'³ ôO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294076737 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129780088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚525800338864128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115721424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129780086 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚525801129511936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7612762468864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129780086 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7612762468864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 # _timestamp‚1.78849583017518e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164296651 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294099755 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74418055 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚512484122451138 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115720891 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚504897994464080 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164319667 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41303367 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚525800338864128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7613535113728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚583292579414528 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164296651 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚575353960232960 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129780088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚525801129511936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚504897977424720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚607 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚512483334763366 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚518187576395264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312W‡–¼ÿ'´ ùO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164319667 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74418057 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚518187594398208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚518187576395264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚575353984173056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7586127987058 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7938813323776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7613535113728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129780086 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚525801129511936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7612762468864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129780086 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7612762468864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 # _timestamp‚1.78849583017518e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164296651 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294099755 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74418055 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚512484122451138 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115720891 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚504897994464080 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164319667 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41303367 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚525800338864128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7613535113728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚583292579414528 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164296651 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚575353960232960 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129780088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚525801129511936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚504897977424720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚607 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚512483334763366 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚518187576395264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294076737 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129780088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚525800338864128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115721424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294115322 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115735768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚504964043860480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7613767214592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7613767214592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294138366 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7614539872768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 trainer/global_step‚7399 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129796962 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41308608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚512550402339004 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294138366 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164318362 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.˜ÃìOfreed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚518255276334592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74427705 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7939857532416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚504964060899840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7587129127144 $ _timestamp‚1.788495831111589e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575429332515840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚575429308575744 _step‚608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115736313 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚525869834210304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _runtime‚7930.752550713 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164341404 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41308065 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚525869043549184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74427703 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7614539872768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7586358478524 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚518255294337536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚518255276334592 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164318362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚512551190026984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129796960 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚518255276334592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74427705 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7939857532416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚504964060899840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7587129127144 $ _timestamp‚1.788495831111589e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575429332515840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚575429308575744 _step‚608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115736313 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚525869834210304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _runtime‚7930.752550713 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164341404 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41308065 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚525869043549184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74427703 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7614539872768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7586358478524 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚518255294337536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚518255276334592 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164318362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚512551190026984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129796960 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294115322 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115735768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚504964043860480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7613767214592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7613767214592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294138366 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7614539872768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 trainer/global_step‚7399 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129796962 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41308608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚512550402339004 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294138366 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0à°æ©P2¸ - _wandbruntime‚7930  _runtime‚7930ñ¡fX¹ Ò %loss/total_loss‚40.38653793622507 .loss/FRt1/assignment_loss‚2.93929926443983 /loss/FRt2/assignment_loss‚3.244781761458426 0loss/FBt4/assignment_loss‚0.1468488855092784 _runtime‚7930.753845122 0loss/FRt4/assignment_loss‚1.0947562394471004 1loss/SRqqt3/assignment_loss‚1.600643917647275 2loss/FBt3/assignment_loss‚0.004480505345954811 /loss/FRt1/detection_loss‚2.1025207669094756 0loss/SRqqt1/detection_loss‚2.193775858481725 /loss/FBt1/detection_loss‚2.7539128959178925 epoch‚6 1loss/SRqqt1/assignment_loss‚1.859338695431749 /loss/FRt3/detection_loss‚2.7156948313117026 /loss/FRt4/detection_loss‚1.5232408066633447 0loss/SRqqt2/detection_loss‚2.108986027779118 1loss/SRqqt3/detection_loss‚1.7109166003086351 /loss/FBt2/detection_loss‚2.6448571562767027 /loss/FBt3/detection_loss‚2.3423566699028013 % _timestamp‚1.7884958311136532e+09 -loss/FRt3/assignment_loss‚2.4960178454717 /loss/FBt4/detection_loss‚0.5511936375319406 _step‚609 2loss/SRqqt4/assignment_loss‚0.8058558717285368 2loss/FBt2/assignment_loss‚0.030291256119357967 .loss/FRt2/detection_loss‚2.366986403269648 1loss/SRqqt4/detection_loss‚0.9848724806850607 trainer/global_step‚7399 2loss/SRqqt2/assignment_loss‚1.5967410120271868 0loss/FBt1/assignment_loss‚0.5681685465606279–Ñ¡À]º × .loss/FRt1/assignment_loss‚2.93929926443983 /loss/FRt2/assignment_loss‚3.244781761458426 0loss/FBt4/assignment_loss‚0.1468488855092784 _runtime‚7930.753845122 0loss/FRt4/assignment_loss‚1.0947562394471004 1loss/SRqqt3/assignment_loss‚1.600643917647275 2loss/FBt3/assignment_loss‚0.004480505345954811 /loss/FRt1/detection_loss‚2.1025207669094756 0loss/SRqqt1/detection_loss‚2.193775858481725 /loss/FBt1/detection_loss‚2.7539128959178925 epoch‚6 1loss/SRqqt1/assignment_loss‚1.859338695431749 /loss/FRt3/detection_loss‚2.7156948313117026 /loss/FRt4/detection_loss‚1.5232408066633447 0loss/SRqqt2/detection_loss‚2.108986027779118 1loss/SRqqt3/detection_loss‚1.7109166003086351 /loss/FBt2/detection_loss‚2.6448571562767027 /loss/FBt3/detection_loss‚2.3423566699028013 % _timestamp‚1.7884958311136532e+09 -loss/FRt3/assignment_loss‚2.4960178454717 /loss/FBt4/detection_loss‚0.5511936375319406 _step‚609 2loss/SRqqt4/assignment_loss‚0.8058558717285368 2loss/FBt2/assignment_loss‚0.030291256119357967 .loss/FRt2/detection_loss‚2.366986403269648 1loss/SRqqt4/detection_loss‚0.9848724806850607 trainer/global_step‚7399 2loss/SRqqt2/assignment_loss‚1.5967410120271868 0loss/FBt1/assignment_loss‚0.5681685465606279 %loss/total_loss‚40.38653793622507áÑÕõœ» ‚(:‘ ÛéÔ­ gpu.0.gpu‚9.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.129599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã×*È„¼ ‚(:ù ÛéÔ¥Êϲ network.sent‚ 1.855243e+07% disk./.usageGB‚1172.6281204223633 disk.drbd1260.out‚1.125 disk.sda.out‚ 558.86328125 disk.dm-0.in‚27096.31640625 disk.sda3.in‚0# memory_percent‚89.7515058517456) proc.memory.availableMB‚ 1679.11328125 network.recv‚ 2.654997e+06) disk./.usagePercent‚78.69605706207227 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚190030.29541015625 disk.dm-1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚27096.31640625" disk.nvme0n1.in‚27096.31640625' disk.nvme0n1.out‚190030.29541015625$ disk.dm-0.out‚190030.29541015625 disk.sda3.out‚ 558.86328125# proc.memory.rssMB‚ 2949.58203125 disk.dm-1.out‚ 558.86328125* proc.memory.percent‚18.002820014953613 cpu‚25.29542355912971 proc.cpu.threads‚174ù°’½ ‚(:‡ êéÔÈæ® gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.862' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3724 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óÔøX¾ ‚(:ô êéÔ•ÆÛ° disk.sda3.in‚0 disk.drbd1260.out‚ 1.12890625 disk.sda.in‚0 disk.sda.out‚ 559.421875 disk.sda1.out‚0$ disk.nvme0c0n1.in‚27096.31640625$ memory_percent‚89.74993228912354) proc.memory.availableMB‚ 1679.37109375) disk./.usagePercent‚78.70151104658174 disk.sda3.out‚ 559.421875 disk.drbd1260.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2949.58203125 proc.cpu.threads‚17% disk./.usageGB‚1172.7093887329102" disk.nvme0n1.in‚27096.31640625" disk.dm-0.out‚190425.416015625 disk.dm-1.out‚ 559.421875* proc.memory.percent‚18.002820014953613 cpu‚25.29500870032749 network.sent‚ 1.8659106e+07 network.recv‚ 2.659239e+06% disk.nvme0n1.out‚190425.416015625 disk.dm-0.in‚27096.31640625 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚190425.416015625Æî5¿¿ ‚(:… ùéÔÄŽË­ gpu.0.gpu‚29.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ijÞG‚À ‚(:÷ ùéÔêý° disk.sda3.out‚ 560.37890625) disk./.usagePercent‚78.70373472802414 disk.sda2.out‚0& disk.nvme0c0n1.out‚190723.12109375 disk.dm-0.in‚27096.31640625$ memory_percent‚89.75303173065186) proc.memory.availableMB‚ 1678.86328125# proc.memory.rssMB‚ 2949.58203125 cpu‚25.29461873949683 disk.sda.out‚ 560.37890625 disk.sda2.in‚0" disk.nvme0n1.in‚27096.31640625 disk.sda3.in‚0 proc.cpu.threads‚17 network.recv‚ 2.663942e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 560.37890625 disk.drbd1260.out‚ 1.12890625 disk.sda.in‚0$ disk.nvme0c0n1.in‚27096.31640625* proc.memory.percent‚18.002820014953613 network.sent‚ 1.8665493e+07% disk./.usageGB‚1172.7425231933594 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚190723.12109375! disk.dm-0.out‚190723.12109375ÃO~ 2Á - _wandbruntime‚7977  _runtime‚7977ýQ›È Â ' lr-AdamW/pg1‚0.00030549277395944214 ' lr-AdamW/pg2‚0.00030549277395944214 trainer/global_step‚7449 % _timestamp‚1.7884958778811321e+09 _runtime‚ 7977.52141608 _step‚610NÌà–Íà Ç _runtime‚ 7977.52141608 _step‚610 ' lr-AdamW/pg1‚0.00030549277395944214 ' lr-AdamW/pg2‚0.00030549277395944214 trainer/global_step‚7449 % _timestamp‚1.7884958778811321e+09â_ÖÕÕ2Ä - _wandbruntime‚7977  _runtime‚7977q¤Ùü'Å öO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130623786 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚521572591364608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165385553 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚521572573361664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚521572591364608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚529235587530752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚7977.522426343 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚587112463150080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚579121377372160 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7990850918400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7663786813952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 296032357 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116468173 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚508201299212720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165385553 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74900457 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚587112228290560 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7663786813952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚521572573361664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130623788 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚579121401312256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚529236378178560 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116468701 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚529235587530752 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚529236378178560 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7663014169088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74900455 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41568244 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚515837514824246 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165408569 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884958778820026e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7663014169088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7991061837824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210919424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚508201316252080 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚611 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚515836727136474 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165408569 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7636198572166 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130623786 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41567718 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296009339ɵ¤Õ]Æ ûO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 UADeviceStatsMonitá…ݤ'or.on_train_batch_start/requested_bytes.all.freed‚515836727136474 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165408569 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7636198572166 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130623786 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41567718 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296009339 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130623786 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚521572591364608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165385553 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚521572573361664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚521572591364608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚529235587530752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚7977.522426343 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚587112463150080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚579121377372160 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7990850918400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7663786813952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 296032357 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116468173 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚508201299212720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165385553 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74900457 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚587112228290560 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7663786813952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚521572573361664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130623788 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚579121401312256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚529236378178560 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116468701 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚529235587530752 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚529236378178560 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7663014169088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74900455 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41568244 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚515837514824246 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165408569 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884958778820026e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7663014169088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7991061837824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210919424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚508201316252080 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚611 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046ã›Ìls2Ç - _wandbruntime‚7978  _runtime‚7978¶+A'È €N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚529305082876928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚529304292215808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130640660 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165407264 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116483080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚521640273300992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74910103 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚579196749655040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7449 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7664791572992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296070968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 296047924 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚587188847923200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41572977 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7636429063632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚515904582400092 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 _step‚612 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41573511 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130640662 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚579196725714944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296070968 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116483616 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚515903794712112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚521640291303936 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7664791572992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 296047924 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130640660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74910105 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7992098268160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7991895750656 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165430306 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚508267365648480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7637199712252 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116483080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚521640273300992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74910103 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚579196749655040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7449 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7664791572992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296070968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 296047924 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚587188847923200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41572977 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7636429063632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚515904582400092 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 _step‚612 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41573511 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130640662 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚579196725714944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296070968 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116483616 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚515903794712112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚521640291303936 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7664791572992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 296047924 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130640660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74910105 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7992098268160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7991895750656 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165430306 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚508267365648480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7637199712252 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130640660ä8 2Ê - _wandbruntime‚7978  _runtime‚7978 ੈ]Ë × /loss/FRt1/assignment_loss‚3.382143681868911 /loss/FRt3/detection_loss‚2.5540337977987346 /loss/SRqqt3/detection_loss‚1.54326556700057 /loss/FRt2/detection_loss‚2.2602627454375246 1loss/SRqqt2/detection_loss‚2.2812174558639526 %loss/total_loss‚42.29256911516846 0loss/FRt3/assignment_loss‚2.8302589058876038 % _timestamp‚1.7884958791212213e+09 _step‚613 1loss/FBt3/assignment_loss‚0.02864447946965735 .loss/FRt1/detection_loss‚2.933371359715238 .loss/FBt1/detection_loss‚3.024853026866913 2loss/SRqqt3/assignment_loss‚1.9034145703089649 2loss/SRqqt4/assignment_loss‚0.9909426622386825 1loss/FBt4/assignment_loss‚0.07639163153988121 /loss/FBt2/detection_loss‚2.8256506522496543 epoch‚6 0loss/FRt2/assignment_loss‚2.9500926180583673 2loss/SRqqt1/assignment_loss‚1.3929439591509956 0loss/FBt2/assignment_loss‚0.3813045647684703 1loss/SRqqt4/detection_loss‚0.9427194903653703 /loss/FBt4/detection_loss‚0.5105227291678194 0loss/FRt4/assignment_loss‚1.0119824462703295 1loss/FBt1/assignment_loss‚0.15775636199396104 /loss/FRt4/detection_loss‚1.3154512091340231 1loss/SRqqt1/detection_loss‚2.6523209441275823 trainer/global_step‚7449 2loss/SRqqt2/assignment_loss‚1.7187108815248524 /loss/FBt3/detection_loss‚2.6243133743604026 _runtime‚7978.761370917FâÐ2YÌ Ü /loss/FRt2/detection_loss‚2.2602627454375246 1loss/SRqqt2/detection_loss‚2.2812174558639526 %loss/total_loss‚42.29256911516846 0loss/FRt3/assignment_loss‚2.8302589058876038 % _timestamp‚1.7884958791212213e+09 _step‚613 1loss/FBt3/assignment_loss‚0.02864447946965735 .loss/FRt1/detection_loss‚2.933371359715238 .loss/FBt1/detection_loss‚3.024853026866913 2loss/SRqqt3/assignment_loss‚1.9034145703089649 2loss/SRqqt4/assignment_loss‚0.9909426622386825 1loss/FBt4/assignment_loss‚0.07639163153988121 /loss/FBt2/detection_loss‚2.8256506522496543 epoch‚6 0loss/FRt2/assignment_loss‚2.9500926180583673 2loss/SRqqt1/assignment_loss‚1.3929439591509956 0loss/FBt2/assignment_loss‚0.3813045647684703 1loss/SRqqt4/detection_loss‚0.9427194903653703 /loss/FBt4/detection_loss‚0.5105227291678194 0loss/FRt4/assignment_loss‚1.0119824462703295 1loss/FBt1/assignment_loss‚0.15775636199396104 /loss/FRt4/detection_loss‚1.3154512091340231 1loss/SRqqt1/detection_loss‚2.6523209441275823 trainer/global_step‚7449 2loss/SRqqt2/a¥3“Š ssignment_loss‚1.7187108815248524 /loss/FBt3/detection_loss‚2.6243133743604026 _runtime‚7978.761370917 /loss/FRt1/assignment_loss‚3.382143681868911 /loss/FRt3/detection_loss‚2.5540337977987346 /loss/SRqqt3/detection_loss‚1.54326556700057åŒþ¡‘Í ‚(:† ˆéÔáúž® gpu.0.gpu‚43.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.023 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A¬EÓ}Î ‚(:ò ˆéÔ¶º¤¯ network.sent‚ 1.8671288e+07 network.recv‚ 2.668264e+06 disk.drbd1260.out‚ 1.1328125 disk.sda.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚190987.17529296875 disk.dm-0.in‚27096.31640625 disk.sda3.out‚ 561.03125 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚27096.31640625' disk.nvme0n1.out‚190987.17529296875$ disk.dm-0.out‚190987.17529296875# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613) disk./.usagePercent‚78.73729296363497 disk.dm-1.in‚0 disk.dm-1.out‚ 561.03125 disk.sda1.out‚0 cpu‚25.29426117693492 proc.cpu.threads‚17% disk./.usageGB‚1173.2425651550293 disk.sda.out‚ 561.03125$ disk.nvme0c0n1.in‚27096.31640625 disk.sda3.in‚0$ memory_percent‚89.75334167480469% proc.memory.availableMB‚ 1678.8125j˜îMÏ ‚(:’ —éÔ–˜Ð­ gpu.0.gpu‚30.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.837' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.167399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘e|†Ð ‚(:û —éÔÆñº®$ disk.dm-0.out‚191082.62060546875* proc.memory.percent‚18.002820014953613% disk./.usageGB‚1173.2425117492676 disk.sda3.out‚ 561.31640625 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 27252.3203125# proc.memory.rssMB‚ 2949.58203125) disk./.usagePercent‚78.73728937952949 disk.sda3.in‚0 disk.dm-1.out‚ 561.31640625 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 27252.3203125! disk.nvme0n1.in‚ 27252.3203125$ memory_percent‚89.75186347961426( proc.memory.availableMB‚ 1679.0546875 disk.sda.out‚ 561.31640625 disk.sda1.in‚0) disk.nvme0c0n1.out‚191082.62060546875' disk.nvme0n1.out‚191082.62060546875 cpu‚25.29387482186436 proc.cpu.threads‚17 network.sent‚ 1.8779245e+07 network.recv‚ 2.673896e+06 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.13281258˜ð¯…Ñ ‚(:ú ¦éÔê´ï¯ disk.dm-1.out‚ 561.93359375 disk.sda.in‚0 disk.sda.out‚ 561.93359375# disk.dm-0.out‚191347.6650390625$ memory_percent‚89.75033760070801 proc.cpu.threads‚17 network.recv‚ 2.677661e+06% disk./.usageGB‚1173.3674507141113 disk.sda3.out‚ 561.93359375 disk.drbd1260.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 27252.3203125( disk.nvme0c0n1.out‚191347.6650390625* proc.memory.percent‚18.002820014953613) disk./.usagePercent‚78.74567413829094 disk.sda1.out‚0& disk.nvme0n1.out‚191347.6650390625 disk.dm-0.in‚ 27252.3203125# proc.memory.rssMB‚ 2949.58203125 cpu‚25.293598895659958 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.13671875 disk.sda1.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 27252.3203125( proc.memory.availableMB‚ 1679.3046875 network.sent‚ 1.8784171e+07ûÒ ‚(:’ ¦éÔ ä¯ gpu.0.gpu‚27.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.554' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.110800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤šœÓ ‚(:‘ µéÔ®ºû® gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚132.71' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.541999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÓA…„Ô ‚(:ù µéÔÜç©° disk.dm-1.out‚ 562.55078125 disk.sda.out‚ 562.55078125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 27252.3203125 disk.dm-0.in‚ 27252.3203125( proc.memory.availableMB‚ 1679.0546875* proc.memory.percent‚18.002820014953613 network.sent‚ 1.879043e+07) disk./.usagePercent‚78.74675474609296 disk.drbd1260.out‚ 1.13671875! disk.nvme0n1.in‚ 27252.3203125 cpu‚25.293299383374986 proc.cpu.threads‚17% disk./.usageGB‚1173.3835525512695 disk.sda3.in‚0 disk.sda3.out‚ 562.55078125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚191853.8056640625# proc.memory.rssMB‚ 2949.58203125 network.recv‚ 2.682211e+06 disk.drbd1260.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚191853.8056640625# disk.dm-0.out‚191853.8056640625$ memory_percent‚89.75186347961426 disk.dm-1.in‚0 ‚ÿC2Õ - _wandbruntime‚8029  _runtime‚8029‹ª¨³ÆÖ À _runtime‚ 8029.62425824 _step‚614 & lr-AdamW/pg1‚0.0002999356286270725 & lr-AdamW/pg2‚0.0002999356286270725 trainer/global_step‚7499 % _timestamp‚1.7884959299838994e+09ö”[ŸË× Å _step‚614 & lr-AdamW/pg1‚0.0002999356286270725 & lr-AdamW/pg2‚0.0002999356286270725 trainer/global_step‚7499 % _timestamp‚1.7884959299838994e+09 _runtime‚ 8029.62425824æ"v³=2Ø - _wandbruntime‚8029  _runtime‚8029 œäêü'Ù öO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117213463 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7685498508862 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884959299849484e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131467486 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚524957570328064 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚532670836197376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166474455 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166474455 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8043024024064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚519190907197354 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚532670836197376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131467488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166497471 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚590932040812032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚8029.625381136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297964959 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚582888818451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚524957588331008 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117212923 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚590931818535424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚532671626845184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚519190119509582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7499 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚524957588331008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚532671626845184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8043222360576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7713265869312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166497471 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7714038514176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚511504621000720 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41830068 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131467488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚524957570328064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538¢Û|Ì(Ú ûO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8043024024064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚519190907197354 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚532670836197376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131467488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166497471 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚590932040812032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚8029.625381136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297964959 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚582888818451456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚524957588331008 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117212923 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚590931818535424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚532671626845184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚519190119509582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7499 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚524957588331008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚532671626845184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8043222360576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7713265869312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166497471 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7714038514176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚511504621000720 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41830068 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131467488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚524957570328064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117213463 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7685498508862 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884959299849484e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131467486 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚524957570328064 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚532670836197376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166474455 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166474455 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540çÙæ‹p2Û - _wandbruntime‚8030  _runtime‚8030“¶?4éÜ ÿM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚525025270267392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166519208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75392503 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7686499648740 $ _timestamp‚1.788495930877648e+09 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298003570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚532740331543552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7715043273216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚532739540882432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117227872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166519208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚591008222128640 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8044275500544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚525025288270336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8044079274496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298003570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7714270615040 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297980526 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131484362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚591008442294784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 _runtime‚8030.518431476 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131484360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7715043273216 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166496166 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚582964142854144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131484360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117228403 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚519257187085220 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚511570704475840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 _step‚616 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166496166 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 297980526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41835369 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚519257974773200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7687270297360 trainer/global_step‚7499 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131484362 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚525025270267392Yjø 'Ý „N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚591008222128640 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8044275500544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚525025288270336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8044079274496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298003570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7714270615040 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297980526 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131484362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚591008442294784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 _runtime‚8030.518431476 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131484360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7715043273216 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166496166 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚582964142854144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131484360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117228403 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚519257187085220 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚511570704475840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 _step‚616 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166496166 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 297980526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41835369 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚519257974773200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7687270297360 trainer/global_step‚7499 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131484362 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚525025270267392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚525025270267392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166519208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75392503 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7686499648740 $ _timestamp‚1.788495930877648e+09 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298003570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚532740331543552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7715043273216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚532739540882432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117227872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166519208èE ¯2Þ - _wandbruntime‚8030  _runtime‚8030Í (žYß Ó 1loss/SRqqt2/detection_loss‚2.3242879509925842 _runtime‚8030.520048132 2loss/SRqqt1/assignment_loss‚0.9820058681070805 2loss/SRqqt4/assignment_loss‚0.9150590405632288 1loss/FBt3/assignment_loss‚0.23701771543856012 1loss/SRqqt4/detection_loss‚0.9035064888974795 /loss/FBt2/detection_loss‚2.3722855831895555 0loss/SRqqt1/detection_loss‚1.954315472394228 1loss/SRqqt3/detection_loss‚1.6417969029529818 /loss/FRt1/detection_loss‚2.3442033513596185 /loss/FRt2/assignment_loss‚3.042911859495299 1loss/FBt4/assignment_loss‚0.11229361079731053 -loss/FRt3/detection_loss‚2.82580224496703 /loss/FRt4/detection_loss‚1.3800175567242232 epoch‚6 -loss/FRt1/assignment_loss‚3.3791354675042 1loss/FBt1/assignment_loss‚0.21416475379373878 /loss/FBt4/detection_loss‚0.4881948179963013 % _timestamp‚1.7884959308798969e+09 1loss/SRqqt3/assignment_loss‚1.539424434736637 0loss/FBt2/assignment_loss‚0.3843371868320641 /loss/FRt2/detection_loss‚2.6265307396518813 0loss/FRt3/assignment_loss‚2.6535812923985143 /loss/FBt1/detection_loss‚2.8511051684617996 /loss/FBt3/detection_loss‚2.1502800226211547 %loss/total_loss‚39.84595264506386 trainer/global_step‚7499 _step‚617 /loss/FRt4/assignment_loss‚1.323501052639701 2loss/SRqqt2/assignment_loss‚1.2001940625486895²ËÑk^à Ø /loss/FRt1/detection_loss‚2.3442033513596185 /loss/FRt2/assignment_loss‚3.042911859495299 1loss/FBt4/assignment_loss‚0.11229361079731053 -loss/FRt3/detection_loss‚2.82580224496703 /loss/FRt4/detection_loss‚1.3800175567242232 epoch‚6 -loss/FRt1/assignment_loss‚3.3791354675042 1loss/FBt1/assignment_loss‚0.21416475379373878 /loss/FBt4/detection_loss‚0.4881948179963013 % _timestamp‚1.7884959308798969e+09 1loss/SRqqt3/assignment_loss‚1.539424434736637 0loss/FBt2/assignment_loss‚0.3843371868320641 /loss/FRt2/detection_loss‚2.6265307396518813 0loss/FRt3/assignment_loss‚2.6535812923985143 /loss/FBt1/detection_loss‚2.8511051684617996 /loss/FBt3/detection_loss‚2.1502800226211547 %loss/total_loss‚39.84595264506386 trainer/global_step‚7499 _step‚617 /loss/FRt4/assignment_loss‚1.323501052639701 2loss/SRqqt2/assignment_loss‚1.2001940625486895 1loss/SRqqt2/detection_loss‚2.3242879509925842 _runtime‚8030.520048132 2loss/SRqqt1/assignment_loss‚0.9820058681070805 2loss/SRqqt4/assignment_loss‚0.9150590405632288 1loss/FBt3/assignment_loss‚0.23701771543856012 1loss/SRqqt4/detection_loss‚0.9035064888974795 /loss/FBt2/detection_loss‚2.3722855831895555 0loss/SRqqt1/detection_loss‚1.954315472394228 1loss/SRqqt3/detection_loss‚1.6417969029529818é:½]’á ‚(:‡ ÄéÔƒŠÿ­ gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.882' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9764 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g~Vâ ‚(:ö Äéԇž¯# disk.dm-0.out‚192046.7744140625 disk.dm-1.in‚0 disk.sda.out‚ 563.48828125% disk./.usageGB‚1173.6334342956543 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚1.140625 disk.sda.in‚0$ memory_percent‚89.75365161895752* proc.memory.percent‚18.002820014953613 proc.cpu.threads‚17& disk.nvme0n1.out‚192046.7744140625 disk.dm-0.in‚ 27252.3203125 disk.sda3.in‚0 disk.dm-1.out‚ 563.48828125 disk.sda1.in‚0) proc.memory.availableMB‚ 1678.76171875 cpu‚25.2929477753856 network.sent‚ 1.8796304e+07 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 27252.3203125( disk.nvme0c0n1.out‚192046.7744140625 disk.sda3.out‚ 563.48828125 disk.sda1.out‚0# proc.memory.rssMB‚ 2949.58203125 network.recv‚ 2.68626e+06) disk./.usagePercent‚78.76352451962336! disk.nvme0n1.in‚ 27252.3203125làã ‚(:… ÓéÔʮڭ gpu.0.gpu‚43.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚162.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,î^nä ‚(:ã ÓéÔÅÍ‘±" disk.nvme0c0n1.in‚ 27385.828125 disk.drbd1260.in‚0* proc.memory.percent‚18.002820014953613 cpu‚25.29261251065373 proc.cpu.threads‚17( disk.nvme0c0n1.out‚192521.3603515625# disk.dm-0.out‚192521.3603515625 disk.sda.in‚0 disk.sda.out‚563.8125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚93.16177368164062 network.recv‚ 2.69213e+06 disk.nvme0n1.in‚ 27385.828125 disk.dm-0.in‚ 27385.828125 disk.sda2.in‚0$ proc.memory.availableMB‚1120.375# proc.memory.rssMB‚ 2949.58203125) disk./.usagePercent‚78.77520588740322% disk./.usageGB‚1173.8074951171875& disk.nvme0n1.out‚192521.3603515625 disk.sda3.in‚0 disk.sda3.out‚563.8125 disk.dm-1.in‚0 disk.dm-1.out‚563.8125 disk.drbd1260.out‚1.140625 network.sent‚ 1.8904183e+07ø6¿œå ‚(:‘ âéÔóâ÷° gpu.0.gpu‚41.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚139.08' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.816000000000003 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žAzæ ‚(:ï âéÔííൠdisk.sda.in‚0' disk.nvme0c0n1.out‚192828.005859375# memory_percent‚93.1617259979248( proc.memory.availableMB‚ 1120.3828125# proc.memory.rssMB‚ 2949.58203125 cpu‚25.292258463559186 network.sent‚ 1.8909197e+07 network.recv‚ 2.695757e+06) disk./.usagePercent‚78.77518003064226% disk./.usageGB‚1173.8071098327637 disk.dm-1.in‚0 disk.sda2.in‚0* proc.memory.percent‚18.002820014953613 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 564.2890625 disk.sda.out‚ 564.2890625" disk.nvme0c0n1.in‚ 27385.828125% disk.nvme0n1.out‚192828.005859375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.1484375 disk.sda1.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 27385.828125 disk.dm-0.in‚ 27385.828125" disk.dm-0.out‚192828.005859375 disk.dm-1.out‚ 564.2890625V\2ç - _wandbruntime‚8081  _runtime‚8081ÃÐQBÇè Á % _timestamp‚1.7884959822500317e+09 _runtime‚8081.890374832 _step‚618 & lr-AdamW/pg1‚0.0002943985281932107 & lr-AdamW/pg2‚0.0002943985281932107 trainer/global_step‚75491+ÔÌé Æ & lr-AdamW/pg2‚0.0002943985281932107 trainer/global_step‚7549 % _timestamp‚1.7884959822500317e+09 _runtime‚8081.890374832 _step‚618 & lr-AdamW/pg1‚0.0002943985281932107ê‚ò"2ê - _wandbruntime‚8081  _runtime‚8081;·ü'ë öO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚514807959828080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 299874543 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7549 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586656211650560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586656235590656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚536106875511808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8095485990400 _runtime‚8081.891771945 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167563357 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚594751721581056 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚514807942788720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75865255 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _step‚619 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132311186 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7736339742382 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117958773 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132311188 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299897561 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42093516 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884959822510972e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚528342567294464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚594751503498752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚536106875511808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚536106084864000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132311188 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚536106084864000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8095291848192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167563357 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42092992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 773556909397045`l`ì ûO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75865255 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _step‚619 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132311186 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7736339742382 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117958773 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132311188 N@DeviceStatsMonitorIðæ¡".on_train_batch_start/allocation.all.allocated‚ 299897561 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42093516 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884959822510972e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚528342567294464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚594751503498752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚536106875511808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚536106084864000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132311188 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚536106084864000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8095291848192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167563357 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42092992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7735569093970 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚514807959828080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 299874543 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7549 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586656211650560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586656235590656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚536106875511808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8095485990400 _runtime‚8081.891771945 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167563357 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚594751721581056 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚514807942788720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115ë /Ð2í - _wandbruntime‚8082  _runtime‚8082ÕñU‰'î €N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚586731559993344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117973041 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299936172 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚522610579458328 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132328062 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚528410285236736 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚594828115935744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 % _timestamp‚1.7884959831727908e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167585068 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75874905 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299913128 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299913128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132328060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚536175580210176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚528410267233792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75874903 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚594827895769600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7736570233848 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚586731583933440 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚522611367146308 _step‚620 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167608110 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7765294973440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7764522315264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚514874009224480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7737340882468 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚514874026263840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117973559 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299936172 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8096532002304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚536175580210176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7764522315264 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42098654 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8096335776256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299936172 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8096532002304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚536175580210176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7764522315264 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42098654 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8096335776256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117973041 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299936172 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚522610579458328 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132328062 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚528410285236736 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚594828115935744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 % _timestamp‚1.7884959831727908e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167585068 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75874905 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299913128 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299913128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132328060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚536175580210176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚528410267233792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75874903 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚594827895769600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7736570233848 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚586731583933440 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚522611367146308 _step‚620ìö#è2ð - _wandbruntime‚8082  _runtime‚8082ß4D}_ñ Ù 2loss/SRqqt2/assignment_loss‚1.9357378573040478 1loss/FBt1/assignment_loss‚0.06589948439573863 3loss/FBt2/assignment_loss‚0.0019080203273157774 1loss/SRqqt4/detection_loss‚1.3071247390545313 .loss/FBt2/detection_loss‚3.346462478240331 /loss/FRt3/assignment_loss‚2.100368731362479 1loss/SRqqt3/assignment_loss‚1.121140419319272 /loss/FRt1/detection_loss‚2.0991397348160925 .loss/FBt1/detection_loss‚2.103224419630491 /loss/FBt4/detection_loss‚0.8578943980922569 %loss/total_loss‚39.37007783243901 0loss/SRqqt1/detection_loss‚2.187494317122868 /loss/FBt3/detection_loss‚2.5779901891946793 3loss/FBt3/assignment_loss‚0.0002651687136676628 /loss/FRt3/detection_loss‚2.3640487181288856 /loss/FRt2/detection_loss‚2.3241320308008233 0loss/SRqqt2/detection_loss‚2.311666464433074 /loss/FRt4/assignment_loss‚1.466271222783969 1loss/FBt4/assignment_loss‚0.05004374459556143 1loss/SRqqt3/detection_loss‚1.5072829698974437 epoch‚6 trainer/global_step‚7549 % _timestamp‚1.7884959831756134e+09 /loss/FRt2/assignment_loss‚3.293113668759664 2loss/SRqqt1/assignment_loss‚1.2555555849203042 2loss/SRqqt4/assignment_loss‚0.9633437128504738 /loss/FRt4/detection_loss‚1.5236723781921542 _runtime‚8082.815959663 _step‚621 0loss/FRt1/assignment_loss‚2.6062973795028834ì92Ñdò Þ epoch‚6 trainer/global_step‚7549 % _timestamp‚1.7884959831756134e+09 /loss/FRt2/assignment_loss‚3.293113668759664 2loss/SRqqt1/assignment_loss‚1.2555555849203042 2loss/SRqqt4/assignment_loss‚0.9633437128504738 /loss/FRt4/detection_loss‚1.5236723781921542 _runtime‚8082.815959663 _step‚621 0loss/FRt1/assignment_loss‚2.6062973795028834 2loss/SRqqt2/assignment_loss‚1.9357378573040478 1loss/FBt1/assignment_loss‚0.06589948439573863 3loss/FBt2/assignment_loss‚0.0019080203273157774 1loss/SRqqt4/detection_loss‚1.3071247390545313 .loss/FBt2/detection_loss‚3.346462478240331 /loss/FRt3/assignment_loss‚2.100368731362479 1loss/SRqqt3/assignment_loss‚1.121140419319272 /loss/FRt1/detection_loss‚2.0991397348160925 .loss/FBt1/detection_loss‚2.103224419630491 /loss/FBt4/detection_loss‚0.8578943980922569 %loss/total_loss‚39.37007783243901 0loss/SRqqt1/detection_loss‚2.187494317122868 /loss/FBt3/detection_loss‚2.5779901891946793 3loss/FBt3/assignment_loss‚0.0002651687136676628 /loss/FRt3/detection_loss‚2.3640487181288856 /loss/FRt2/detection_loss‚2.3241320308008233 0loss/SRqqt2/detection_loss‚2.311666464433074 /loss/FRt4/assignment_loss‚1.466271222783969 1loss/FBt4/assignment_loss‚0.05004374459556143 1loss/SRqqt3/detection_loss‚1.5072829698974437플Ew’ó ‚(:‡ ñéÔ’û­ gpu.0.gpu‚18.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ ßLVô ‚(:ó ñéÔó–ñ disk.sda1.out‚0 disk.sda2.out‚0 cpu‚25.291932454486314 disk.sda3.out‚ 564.73828125 disk.dm-1.out‚ 564.73828125' disk.nvme0c0n1.out‚193020.595703125 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 564.73828125 disk.sda2.in‚0* proc.memory.percent‚18.002820014953613 network.sent‚ 1.8915545e+07 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.1484375$ memory_percent‚93.16344261169434# proc.memory.rssMB‚ 2949.58203125 proc.cpu.threads‚17 network.recv‚ 2.700779e+06) disk./.usagePercent‚78.80736068971133týÀå(% disk./.usageGB‚1174.2866249084473 disk.nvme0n1.in‚ 27385.828125% disk.nvme0n1.out‚193020.595703125" disk.dm-0.out‚193020.595703125 disk.sda1.in‚0( proc.memory.availableMB‚ 1120.1015625" disk.nvme0c0n1.in‚ 27385.828125 disk.dm-0.in‚ 27385.828125 disk.sda.in‚0šœw’õ ‚(:‡ €‘éÔÁ gpu.0.gpu‚12.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚126.112' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2224 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚AŸxö ‚(:í €‘éÔ®Šª±$ memory_percent‚93.16368103027344# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613 disk.sda1.in‚0 disk.dm-0.in‚ 27385.828125 disk.dm-1.in‚0% proc.memory.availableMB‚ 1120.0625 cpu‚25.29161243797353 proc.cpu.threads‚17 network.recv‚ 2.706774e+06% disk./.usageGB‚1174.3132514953613 disk.sda.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 27385.828125 disk.nvme0n1.in‚ 27385.828125' disk.nvme0n1.out‚193412.01708984375$ disk.dm-0.out‚193412.01708984375 network.sent‚ 1.9024053e+07) disk./.usagePercent‚78.80914762230033 disk.sda2.out‚0) disk.nvme0c0n1.out‚193412.01708984375 disk.sda3.in‚0 disk.sda3.out‚ 565.34375 disk.dm-1.out‚ 565.34375 disk.drbd1260.out‚ 1.15234375 disk.sda.out‚ 565.34375 disk.sda2.in‚0 disk.drbd1260.in‚0mäu÷ ‚(:… ‘éÔ‚õ® gpu.0.gpu‚31.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿæh‰ø ‚(:þ ‘éÔ—„û±) disk./.usagePercent‚78.80276279439637 disk.sda1.in‚0' disk.nvme0n1.out‚193796.81005859375$ disk.dm-0.out‚193796.81005859375 disk.dm-1.out‚ 566.0546875 disk.drbd1260.out‚ 1.15234375 disk.sda.in‚0 disk.sda.out‚ 566.0546875 network.sent‚ 1.9031209e+07% disk./.usageGB‚1174.2181129455566 disk.sda2.in‚0 disk.dm-0.in‚27504.80859375 disk.sda3.in‚0$ memory_percent‚93.16213130950928) proc.memory.availableMB‚ 1120.31640625# proc.memory.rssMB‚ 2949.58203125 network.recv‚ 2.710268e+06 disk.sda1.out‚0$ disk.nvme0c0n1.in‚27504.80859375 disk.sda3.out‚ 566.0546875 cpu‚25.291257678211384 proc.cpu.threads‚17 disk.sda2.out‚0) disk.nvme0c0n1.out‚193796.81005859375" disk.nvme0n1.in‚27504.80859375 disk.dm-1.in‚0 disk.drbd1260.in‚0* proc.memory.percent‚18.002820014953613ü›Yc’ù ‚(:‡ ž‘éÔ‚ê®­ gpu.0.gpu‚48.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.873' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9746 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê©¥C„ú ‚(:ù ž‘éÔÁ½É± disk.dm-0.in‚27504.80859375$ memory_percent‚93.16363334655762 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚78.80854600459486 disk.sda3.out‚ 566.65234375 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚194061.1298828125 network.sent‚ 1.9034329e+07# disk.dm-0.out‚194061.1298828125# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613 proc.cpu.threads‚17 network.recv‚ 2.713397e+06$ disk./.usageGB‚1174.304286956787 disk.sda3.in‚0 disk.drbd1260.out‚1.15625$ disk.nvme0c0n1.in‚27504.80859375( proc.memory.availableMB‚ 1120.0703125 cpu‚25.290941406511283 disk.dm-1.in‚0 disk.dm-1.out‚ 566.65234375 disk.sda.out‚ 566.65234375 disk.sda2.in‚0" disk.nvme0n1.in‚27504.80859375& disk.nvme0n1.out‚194061.1298828125E]ž2û - _wandbruntime‚8131  _runtime‚8131Æ*†âÉü à ' lr-AdamW/pg1‚0.00028888262658128806 ' lr-AdamW/pg2‚0.00028888262658128806 trainer/global_step‚7599 % _timestamp‚1.7884960320074515e+09 _runtime‚8131.647831238 _step‚622ò‚~.Îý È ' lr-AdamW/pg1‚0.00028888262658128806 ' lr-AdamW/pg2‚0.00028888262658128806 trainer/global_step‚7599 % _timestamp‚1.7884960320074515e+09 _runtime‚8131.647831238 _step‚622îÙĤA2þ - _wandbruntime‚8131  _runtime‚8131(‘xIü'ÿ öO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133154888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚518111281616080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118703440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚598571148989440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚539541333530624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7814541914624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7814541914624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168675275 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7813769269760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚518111264576720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _runtime‚8131.649077707 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168652259 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133154888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531727582263808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118702909 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590423628789760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 301830163 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133154886 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42355254 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7786410327490 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76347655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 trainer/global_step‚7599 % _timestamp‚1.7884960320085034e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚525897691943570 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168652259 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7785639679078 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚531727582263808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531727564260864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚525896904255798 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 301807145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚539541333530624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚539542124178432 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133154886 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8147720633344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 _step‚623:‚e(€!ûO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚539542124178432 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133154886 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8147720633344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 _step‚623 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133154888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚518111281616080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118703440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚598571148989440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚539541333530624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7814541914624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7814541914624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168675275 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7813769269760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚518111264576720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _runtime‚8131.649077707 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168652259 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133154888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531727582263808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118702909 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590423628789760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 301830163 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133154886 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42355254 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7786410327490 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76347655 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 trainer/global_step‚7599 % _timestamp‚1.7884960320085034e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚525897691943570 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168652259 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7785639679078 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚531727582263808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531727564260864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚525896904255798 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 301807145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚539541333530624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489ïaåŒ2!- _wandbruntime‚8132  _runtime‚8132/ÊÏ ‚!€N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚598647768337408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133171760 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168697012 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118717685 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚518177348051840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚518177331012480 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7815546673664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7814774015488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42360918 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590498977132544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 301868774 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133171762 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76357305 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884960328680053e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚531795282203136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168697012 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8148564747264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚539610828876800 ZMDeviceStatsMonitor.on_train_batch_end/inacti­€ú ve_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 trainer/global_step‚7599 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7786640818956 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚539610038215680 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168673970 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76357303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42360382 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8148767264768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 301868774¯¾x 'ƒ!…N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7787411467576 _step‚624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598647541879808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118717685 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚518177348051840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚518177331012480 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7815546673664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7814774015488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42360918 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590498977132544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 301868774 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133171762 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76357305 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884960328680053e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚531795282203136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168697012 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8148564747264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚539610828876800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 trainer/global_step‚7599 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7786640818956 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚539610038215680 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168673970 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76357303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42360382 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8148767264768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 301868774 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚598647768337408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133171760 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168697012 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542976582197248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169741161 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76830057 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 _step‚627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76830055 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133998586 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 % _timestamp‚1.7884960817200758e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚542977372845056 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚602391010954240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303762765 trainer/global_step‚7649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7864020969984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚542976582197248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚529251084316678 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚521414586364720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 303739747 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8199744845824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚535112561227264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚535112579230208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚529250296628906 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7835710264186 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7864793614848ç^uV ’!úO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚535112579230208 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133998586 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7864793614848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7864793614848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542976582197248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169741161 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76830057 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 _step‚627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76830055 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133998586 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_tÇäªrain_batch_start/reserved_bytes.large_pool.current‚ 11129585664 % _timestamp‚1.7884960817200758e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚542977372845056 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚602391010954240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303762765 trainer/global_step‚7649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7864020969984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚542976582197248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚529251084316678 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚521414586364720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 303739747 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8199744845824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚535112561227264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚535112579230208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚529250296628906 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7835710264186 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76839705 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7836711404064 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169762872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76839703 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169785914 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119464288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 8182.21217385 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303801376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7865798373888 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119463746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚535180261166592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 % _timestamp‚1.7884960825710845e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚543046077543424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚543046077543424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8200990000128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8200787482624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134015460 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚521480652800480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚535180279169536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚535180279169536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7865025715712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚594266394271744 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134015462 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚602467408211968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚529317364204544 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169762872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚535180261166592 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚594266418211840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚628 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42624583 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚7649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303778332 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7865025715712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540jìà’ '•!„N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚535180261166592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 % _timestamp‚1.7884960825710845e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚543046077543424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚543046077543424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8200990000128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8200787482624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134015460 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚521480652800480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚535180279169536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚535180279169536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7865025715712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚594266394271744 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134015462 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚602467408211968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚529317364204544 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169762872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚535180261166592 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚594266418211840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚628 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42624583 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚7649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303778332 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7865025715712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 303778332 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76839705 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7836711404064 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169762872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76839703 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169785914 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119464288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 8182.21217385 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303801376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7865798373888 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119463746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8251888823296 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134842288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134842286 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7914272670208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42881607 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42882152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚524717925192080 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7886551497706 _runtime‚8232.410314753 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7915045315072 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7915045315072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚597958487008256 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚524717908152720 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120194609 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚546412621511680 % _timestamp‚1.7884961327693837e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134842288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚546412621511680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚538497576196608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚606210351891456 trainer/global_step‚7699 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚538497558193664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170853079 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚597958463068160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7885780849294 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚546411830863872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170830063 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚546411830863872 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8252089256960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚631 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120194062 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚532603689002014 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚538497558193664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚532604476689786 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚597958487008256 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚524717908152720 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120194609 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚546412621511680 % _timestamp‚1.7884961327693837e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134842288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚546412621511680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚538497576196608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚606210351891456 trainer/global_step‚7699 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚538497558193664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170853079 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚597958463068160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7885780849294 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚546411830863872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170830063 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚546411830863872 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8252089256960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚631 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120194062 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚532603689002014 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚538497558193664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚532604476689786 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8251888823296 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134842288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992÷œ‘F2¥!- _wandbruntime‚8233  _runtime‚82338hó5'¦!€N EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7916050074112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134859162 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120209478 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77322103 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305733978 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚538565276135936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170874816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚598033811410944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚606286965587968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134859162 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 213003264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 _step‚632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42886841 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8253130236928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7915277415936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236943360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8252917233664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚532671544265632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7887552637792 _runtime‚8233.581966919 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120208944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42887373 % _timestamp‚1.7884961339404166e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305733978 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77322105 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134859160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚546480535548928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚606286728644608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚538565258132992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134859160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7916050074112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170851774 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚546481326210048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚538565276135936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚538565258132992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7915277415936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚524783991627840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚524783974588480 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7886781989172 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305710934 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7916050074112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134859162 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120209478 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77322103 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305733978 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚538565276135936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170874816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚598033811410944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚606286965587968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134859162 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 213003264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 _step‚632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42886841 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8253130236928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7915277415936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236943360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8252917233664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚532671544265632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7887552637792 _runtime‚8233.581966919 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120208944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42887373 % _timestamp‚1.7884961339404166e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305733978 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77322105 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134859160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚546480535548928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚606286728644608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚538565258132992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134859160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7916050074112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170851774 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚546481326210048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚538565276135936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚538565258132992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7915277415936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚524783991627840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚524783974588480 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7886781989172 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305710934 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120940539 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚541882555160064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7964524370432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚528021246980080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135685986 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚635 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43145684 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77794855 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77794857 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚541882555160064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610030092822528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚601725880207360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8304402563072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7936622082814 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135685988 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚601725904147456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚549847870178304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚541882573163008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135685986 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171941981 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307604951 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚549847079530496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚528021229940720 % _timestamp‚1.7884961852557917e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549847079530496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307627969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43146212 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43145684 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77794855 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMoZ¥-š¯nitor.on_train_batch_start/active.large_pool.peak‚3125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77794857 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚541882555160064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610030092822528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚601725880207360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8304402563072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7936622082814 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135685988 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚601725904147456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚549847870178304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚541882573163008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135685986 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171941981 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307604951 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚549847079530496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚528021229940720 % _timestamp‚1.7884961852557917e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549847079530496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307627969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43146212 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120940539ûÎüµ¦2¹!- _wandbruntime‚8285  _runtime‚8285Ÿ±Mð'º!€N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚601801228550144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚549916574876672 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171940676 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7965529116160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚541950273102336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 307643536 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 _step‚636 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77804503 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚601801252490240 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307666580 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171963718 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8305255727104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚536024936638740 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7966301774336 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120956004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528087313415840 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307643536 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120955454 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚549915784215552 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171940676 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884961861216724e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135702862 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43151499 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43150951 _runtime‚8285.762641007 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚541950273102336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7966301774336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚541950255099392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307666580 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77804505 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135702860 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610106708637696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528087296376480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7936852574280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚541950255099392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7966301774336 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120956004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528087313415840 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307643536 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120955454 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚549915784215552 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171940676 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884961861216724e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135702862 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43151499 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43150951 _runtime‚8285.762641007 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚541950273102336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7966301774336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚541950255099392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307666580 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77804505 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135702860 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610106708637696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528087296376480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7936852574280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚541950255099392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136529688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7986692667922 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545267552126464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43410078 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 _step‚639 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78277257 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚553283118844928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539310473748230 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173007867 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121687876 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚531324568768080 _runtime‚8334.635312289 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553282328197120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539311261436002 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211790848 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136529686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚545267570129408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8015548715520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173030883 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8014776070656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8356911766016 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8015548715520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8014776070656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚605493321286656 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136529688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309537553 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7985922019510 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43410619 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187850752 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78277255 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8356723915264ßÁý(Ê!ûO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43410619 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187850752 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78277255 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8356723915264 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 % _timestamp‚1.7884962349942882e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚545267552126464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136529688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7986692667922 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545267552126464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43410078 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 _step‚639 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78277257 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚553283118844928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539310473748230 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173007867 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121687876 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚531324568768080 _runtime‚8334.635312289 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553282328197120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539311261436002 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211790848 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136529686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚545267570129408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8015548715520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173030883 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8014776070656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8356911766016 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8015548715520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8014776070656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚605493321286656 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136529688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309537553 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7985922019510 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016ÿñá[µ2Ë!- _wandbruntime‚8335  _runtime‚83355Å'Ì!€N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136546560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78286903 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78286905 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚605568645689344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7986923159388 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚613926417658368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553351032882176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309576138 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121702273 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7987693808008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8357966097920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539378329011848 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613926635727360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173029578 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8015780816384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309599182 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚557 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43415925 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43415370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136546562 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553351823543296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8016553474560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚531390635203840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚555 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194128896 _runtime‚8335.709288932 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136546560 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309599182 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚557 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43415925 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43415370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136546562 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553351823543296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8016553474560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚531390635203840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚555 [MDeviceStatsMonitor.on_train_batch_end/inactive_split,îð!_bytes.small_pool.current‚ 194128896 _runtime‚8335.709288932 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136546560 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78286903 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78286905 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚605568645689344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7986923159388 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚613926417658368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553351032882176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309576138 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121702273 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7987693808008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8357966097920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539378329011848 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613926635727360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173029578 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8015780816384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992€mxðj2Î!- _wandbruntime‚8335  _runtime‚8335ø‘µ•\Ï!Ö 2loss/SRqqt3/assignment_loss‚1.4167956398440558 1loss/SRqqt1/detection_loss‚2.2651553479107944 0loss/SRqqt2/detection_loss‚2.733656461238861 0loss/FRt2/assignment_loss‚2.1763332678425695 1loss/FBt1/assignment_loss‚0.14921695170712404 .loss/FRt1/detection_loss‚2.284098332611526 .loss/FBt2/detection_loss‚2.390664917581222 2loss/SRqqt2/assignment_loss‚1.4718471866846086 2loss/SRqqt4/assignment_loss‚1.1051734971654852 1loss/SRqqt4/detection_loss‚0.8504270612057753 %loss/total_loss‚38.90058793903012 1loss/FBt4/assignment_loss‚0.07348932593626675 0loss/FRt4/assignment_loss‚1.2452475449868612 2loss/FBt3/assignment_loss‚0.057855859042047084 /loss/FRt2/detection_loss‚2.4590373992394916 /loss/FRt4/detection_loss‚1.3206723758318861 0loss/SRqqt3/detection_loss‚1.681161088378806 /loss/FRt1/assignment_loss‚2.900329508432528 /loss/FRt3/assignment_loss‚2.792133939266205 2loss/SRqqt1/assignment_loss‚1.2974036394646673 /loss/FBt1/detection_loss‚2.8043326302008196 /loss/FBt3/detection_loss‚2.2177311062812803 _runtime‚8335.710602357 0loss/FBt2/assignment_loss‚0.0403884111297978 .loss/FRt3/detection_loss‚2.633315497636795 epoch‚6 % _timestamp‚1.7884962360699422e+09 _step‚641 /loss/FBt4/detection_loss‚0.5341209494106379 trainer/global_step‚7799,…0?aÐ!Û /loss/FRt2/detection_loss‚2.4590373992394916 /loss/FRt4/detection_loss‚1.3206723758318861 0loss/SRqqt3/detection_loss‚1.681161088378806 /loss/FRt1/assignment_loss‚2.900329508432528 /loss/FRt3/assignment_loss‚2.792133939266205 2loss/SRqqt1/assignment_loss‚1.2974036394646673 /loss/FBt1/detection_loss‚2.8043326302008196 /loss/FBt3/detection_loss‚2.2177311062812803 _runtime‚8335.710602357 0loss/FBt2/assignment_loss‚0.0403884111297978 .loss/FRt3/detection_loss‚2.633315497636795 epoch‚6 % _timestamp‚1.7884962360699422e+09 _step‚641 /loss/FBt4/detection_loss‚0.5341209494106379 trainer/global_step‚7799 2loss/SRqqt3/assignment_loss‚1.4167956398440558 1loss/SRqqt1/detection_loss‚2.2651553479107944 0loss/SRqqt2/detection_loss‚2.733656461238861 0loss/FRt2/assignment_loss‚2.1763332678425695 1loss/FBt1/assignment_loss‚0.14921695170712404 .loss/FRt1/detection_loss‚2.284098332611526 .loss/FBt2/detection_loss‚2.390664917581222 2loss/SRqqt2/assignment_loss‚1.4718471866846086 2loss/SRqqt4/assignment_loss‚1.1051734971654852 1loss/SRqqt4/detection_loss‚0.8504270612057753 %loss/total_loss‚38.90058793903012 1loss/FBt4/assignment_loss‚0.07348932593626675 0loss/FRt4/assignment_loss‚1.2452475449868612 2loss/FBt3/assignment_loss‚0.057855859042047084å˜`Á’Ñ!‚(:‡ ð’éÔÃ gpu.0.gpu‚50.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9564 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r…!ÅpÒ!‚(:å ð’éÔ™±ª³ network.recv‚ 2.7776e+06$ disk./.usageGB‚1175.195053100586 disk.sda2.out‚0' disk.nvme0c0n1.out‚199296.736328125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.18359375 network.sent‚ 1.9523222e+07 disk.dm-0.in‚ 27991.0625 disk.sda3.out‚ 573.796875 disk.sda.in‚0 disk.sda.out‚ 573.796875# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613 proc.cpu.threads‚17 disk.sda1.out‚0 disk.nvme0n1.in‚ 27991.0625% disk.nvme0n1.out‚199296.736328125 disk.dm-1.out‚ 573.796875) proc.memory.availableMB‚ 1120.07421875 cpu‚25.286583863275066) disk./.usagePercent‚78.86832606790777 disk.sda1.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 27991.0625" disk.dm-0.out‚199296.736328125# memory_percent‚93.1636095046997ˆÓ!‚(:… ÿ’éÔ°±Á­ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÜ ±pÔ!‚(:å ÿ’éÔÁ’ï° disk.sda2.in‚0* proc.memory.percent‚18.002820014953613 cpu‚25.28629716238643 proc.cpu.threads‚17( disk./.usagePercent‚78.8666200336996% disk./.usageGB‚1175.1696319580078 disk.nvme0c0n1.in‚ 27991.0625% disk.nvme0n1.out‚199877.505859375 disk.sda3.out‚ 574.671875 disk.sda1.out‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 1119.796875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.18359375$ memory_percent‚93.16530227661133 network.recv‚ 2.783029e+06' disk.nvme0c0n1.out‚199877.505859375 disk.nvme0n1.in‚ 27991.0625" disk.dm-0.out‚199877.505859375 disk.sda.in‚0 disk.sda.out‚ 574.671875 disk.sda1.in‚0# proc.memory.rssMB‚ 2949.58203125 network.sent‚ 1.9632908e+07 disk.dm-0.in‚ 27991.0625 disk.dm-1.out‚ 574.671875:œÕ!‚(:‘ Ž“éԶТ® gpu.0.gpu‚7.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.471' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.294200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0özWž‚Ö!‚(:÷ Ž“éÔ•ª®² disk.dm-0.in‚28108.67578125 network.sent‚ 1.9637931e+07$ disk./.usageGB‚1175.193042755127 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 574.9765625' proc.memory.availableMB‚ 1120.046875# proc.memory.rssMB‚ 2949.58203125 cpu‚25.286045908892405 disk.drbd1260.out‚1.1875 disk.sda1.in‚0$ disk.nvme0c0n1.in‚28108.67578125" disk.nvme0n1.in‚28108.67578125 disk.dm-1.in‚0 disk.dm-1.out‚ 574.9765625$ memory_percent‚93.16377639770508* proc.memory.percent‚18.002820014953613 disk.sda.out‚ 574.9765625) disk.nvme0c0n1.out‚200194.19580078125$ disk.dm-0.out‚200194.19580078125 proc.cpu.threads‚17 network.recv‚ 2.786696e+06) disk./.usagePercent‚78.86819115193724 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚200194.19580078125“¿ˆs2×!- _wandbruntime‚8384  _runtime‚8384PD8ÇØ!Á trainer/global_step‚7849 % _timestamp‚1.7884962848985817e+09 _runtime‚8384.539015445 _step‚642 & lr-AdamW/pg1‚0.0002616611655371431 & lr-AdamW/pg2‚0.0002616611655371431ò|ÉRÌÙ!Æ & lr-AdamW/pg2‚0.0002616611655371431 trainer/global_step‚7849 % _timestamp‚1.7884962848985817e+09 _runtime‚8384.539015445 _step‚642 & lr-AdamW/pg1‚0.0002616611655371431‚KÄ 2Ú!- _wandbruntime‚8384  _runtime‚8384Ù6} ü'Û!öO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174096769 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚553 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚617669905736704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8035992604618 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43675495 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8065800415744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 311493173 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311470155 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚548652549092864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542664653809110 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚556718367511552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚548652567095808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8065800415744 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122435150 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137373386 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚609260738425856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚548652567095808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8036763253030 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137373388 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174119785 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78759657 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174119785 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556717576863744 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884962848999555e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78759655 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚609260714485760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚556717576863744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚534627890556080 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7849 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122435703 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137373388 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚548652549092864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8065027770880 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8409383296000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚534627873516720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _runtime‚8384.540528326›—I¦žÜ!ûO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43676046 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚542663866121338 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚643 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8065027770880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8409191250944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚617670121721856 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137373386 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174096769 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884962848999555e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78759655 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚609260714485760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚556717576863744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚534627890556080 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7849 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122435703 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137373388 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚548652549092864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8065027770880 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8409383296000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚534627873516720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _runtime‚8384.540528326 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174096769 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚553 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚617669905736704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8035992604618 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43675495 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8065800415744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 311493173 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311470155 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚548652549092864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542664653809110 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 101763482l_’c5 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚556718367511552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚548652567095808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8065800415744 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122435150 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137373386 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚609260738425856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚548652567095808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8036763253030 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137373388 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174119785 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78759657 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174119785 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556717576863744 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311531784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚534693939952480 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174118480 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137390260 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311508740 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137390260 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788496285848087e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚556787072209920 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚542730933696976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122450627 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚548720249032192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8066032516608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137390262 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542731721384956 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 trainer/global_step‚7849 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 _runtime‚8385.489368761 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137390262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78769303 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8410429033472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚548720267035136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8036993744496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚548720267035136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚557 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 _step‚644 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚556787072209920 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174141522 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8066805174784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8066805174784 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8037764393116 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚555 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8066032516608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174141522 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174118480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122450070nùù« 'á!„N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311531784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚534693939952480 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174118480 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137390260 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311508740 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137390260 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788496285848087e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚556787072209920 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚542730933696976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122450627 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚548720249032192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8066032516608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137390262 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542731721384956 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 trainer/global_step‚7849 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 _runtime‚8385.489368761 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137390262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78769303 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8410429033472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚548720267035136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8036993744496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚548720267035136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚557 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 _step‚644 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚556787072209920 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174141522 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8066805174784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8066805174784 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8037764393116 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚555 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8066032516608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174141522 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174118480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122450070 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43681322 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚617746515802112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚617746291441664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 311508740 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78769305 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992„OòÌ2â!- _wandbruntime‚8385  _runtime‚8385ƒŸ¥ì]ã!× 0loss/FRt2/assignment_loss‚3.0698289803096226 2loss/SRqqt3/assignment_loss‚0.8758431741157908 1loss/FBt2/assignment_loss‚0.30780559976702193 2loss/SRqqt2/assignment_loss‚1.5924757760949433 /loss/FBt2/detection_loss‚2.6325163543224335 /loss/FBt3/detection_loss‚2.1220643758773803 /loss/FBt4/detection_loss‚0.6948076096115752 _step‚645 1loss/FBt4/assignment_loss‚0.06650470836223121 .loss/FRt1/detection_loss‚2.340850197011605 /loss/FBt1/detection_loss‚2.4271938460213796 0loss/FRt4/assignment_loss‚1.3107933861310366 0loss/SRqqt1/detection_loss‚2.268807074853352 1loss/SRqqt3/detection_loss‚1.4264608301093866 % _timestamp‚1.7884962858514352e+09 /loss/FRt4/detection_loss‚1.4559972281949443 epoch‚6 /loss/FRt1/assignment_loss‚3.748858427007993 2loss/SRqqt4/assignment_loss‚0.9763999164570123 /loss/FRt3/assignment_loss‚2.143176582726565 1loss/FBt1/assignment_loss‚0.30698932940140367 0loss/SRqqt2/detection_loss‚2.380314363883092 _runtime‚8385.491691378 2loss/SRqqt1/assignment_loss‚1.7428563838558537 0loss/FBt3/assignment_loss‚0.1519442400058324 /loss/FRt2/detection_loss‚2.2611783349965533 /loss/FRt3/detection_loss‚2.8847237595103006 1loss/SRqqt4/detection_loss‚0.7005338807674971 %loss/total_loss‚39.88892435939481 trainer/global_step‚7849&_§£bä!Ü 2loss/SRqqt1/assignment_loss‚1.7428563838558537 0loss/FBt3/assignment_loss‚0.1519442400058324 /loss/FRt2/detection_loss‚2.2611783349965533 /loss/FRt3/detection_loss‚2.8847237595103006 1loss/SRqqt4/detection_loss‚0.7005338807674971 %loss/total_loss‚39.88892435939481 trainer/global_step‚7849 0loss/FRt2/assignment_loss‚3.0698289803096226 2loss/SRqqt3/assignment_loss‚0.8758431741157908 1loss/FBt2/assignment_loss‚0.30780559976702193 2loss/SRqqt2/assignment_loss‚1.5924757760949433 /loss/FBt2/detection_loss‚2.6325163543224335 /loss/FBt3/detection_loss‚2.1220643758773803 /loss/FBt4/detection_loss‚0.6948076096115752 _step‚645 1loss/FBt4/assignment_loss‚0.06650470836223121 .loss/FRt1/detection_loss‚2.340850197011605 /loss/FBt1/detection_loss‚2.4271938460213796 0loss/FRt4/assignment_loss‚1.3107933861310366 0loss/SRqqt1/detection_loss‚2.268807074853352 1loss/SRqqt3/detection_loss‚1.4264608301093866 % _timestamp‚1.7884962858514352e+09 /loss/FRt4/detection_loss‚1.4559972281949443 epoch‚6 /loss/FRt1/assignment_loss‚3.748858427007993 2loss/SRqqt4/assignment_loss‚0.9763999164570123 /loss/FRt3/assignment_loss‚2.143176582726565 1loss/FBt1/assignment_loss‚0.30698932940140367 0loss/SRqqt2/detection_loss‚2.380314363883092 _runtime‚8385.491691378…žÐCÑ’å!‚(:‡ ¬“éÔú÷ë± gpu.0.gpu‚20.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚138.059' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6118 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mò‡æ!‚(:ü ¬“éÔœ«¬´ network.recv‚ 2.797533e+06 disk.sda3.out‚ 575.8046875 disk.sda.in‚0 disk.sda.out‚ 575.8046875) disk.nvme0c0n1.out‚200668.06396484375" disk.nvme0n1.in‚28108.67578125# proc.memory.rssMB‚ 2949.58203125 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚28108.67578125 network.sent‚ 1.9739478e+07) disk./.usagePercent‚78.86832222779476 disk.dm-1.out‚ 575.8046875 disk.sda2.out‚0' disk.nvme0n1.out‚200668.06396484375) proc.memory.availableMB‚ 1118.74609375* proc.memory.percent‚18.002820014953613 cpu‚25.28540466889498$ disk./.usageGB‚1175.194995880127 disk.sda3.in‚0 disk.drbd1260.out‚ 1.19140625 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚28108.67578125$ disk.dm-0.out‚200668.06396484375$ memory_percent‚93.17171573638916·ß˜Àç!‚(:’ »“éÔæÄݬ gpu.0.gpu‚35.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.454' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.890800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e,–€è!‚(:õ »“éÔÅîé¯( disk./.usagePercent‚78.8429933543721 disk.sda.out‚ 576.28125 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.285105306998037 proc.cpu.threads‚17% disk./.usageGB‚1174.8175773620605$ disk.nvme0c0n1.in‚28108.67578125 disk.dm-0.in‚28108.67578125$ disk.dm-0.out‚200924.29052734375 disk.dm-1.out‚ 576.28125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.983666e+07' disk.nvme0n1.out‚200924.29052734375 disk.sda3.in‚0 disk.sda3.out‚ 576.28125 disk.drbd1260.out‚ 1.19140625$ memory_percent‚93.17059516906738( proc.memory.availableMB‚ 1118.9296875# proc.memory.rssMB‚ 2949.58203125 network.recv‚ 2.802411e+06) disk.nvme0c0n1.out‚200924.29052734375" disk.nvme0n1.in‚28108.67578125 disk.dm-1.in‚0 disk.drbd1260.in‚0* proc.memory.percent‚18.002820014953613fŸŸ‘é!‚(:† Ê“éÔ›‚£® gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚178.875' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À}ˆš}ê!‚(:ò Ê“éÔñÐÖ¯ disk.dm-1.in‚0 disk.sda.out‚ 576.94140625 disk.sda2.out‚0# memory_percent‚93.1694507598877 disk.sda3.in‚0 network.sent‚ 1.9843568e+07% disk./.usageGB‚1174.5466423034668$ disk.nvme0c0n1.in‚28214.15234375% disk.nvme0c0n1.out‚201326.0078125" disk.nvme0n1.in‚28214.15234375 disk.dm-0.out‚201326.0078125 disk.drbd1260.in‚0) disk./.usagePercent‚78.82481067526021 disk.drbd1260.out‚ 1.1953125 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2949.58203125 cpu‚25.284779940483055 proc.cpu.threads‚17 disk.dm-1.out‚ 576.94140625 network.recv‚ 2.805604e+06 disk.dm-0.in‚28214.18359375 disk.sda1.in‚0( proc.memory.availableMB‚ 1119.1171875* proc.memory.percent‚18.002820014953613# disk.nvme0n1.out‚201326.0078125 disk.sda3.out‚ 576.94140625¿Dã@2ë!- _wandbruntime‚8432  _runtime‚8432XˆâÇì!Á ' lr-AdamW/pg2‚0.00025629642513883395 trainer/global_step‚7899 % _timestamp‚1.7884963330413568e+09 _runtime‚ 8432.6818436 _step‚646 ' lr-AdamW/pg1‚0.00025629642513883395 íˆ,Ìí!Æ _runtime‚ 8432.6818436 _step‚646 ' lr-AdamW/pg1‚0.00025629642513883395 ' lr-AdamW/pg2‚0.00025629642513883395 trainer/global_step‚7899 % _timestamp‚1.7884963330413568e+09†#ÝHŸ2î!- _wandbruntime‚8432  _runtime‚8432¹7[3†ï!öO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138217088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 1752086¯N“v&87 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚560153616178176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43940773 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚613028131624960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚537931212344080 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138217088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _step‚647 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175208687 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313402757 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚560152825530368 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123183384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79242057 _runtime‚8432.682883139 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8115279471104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚560152825530368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313425775 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138217086 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚560153616178176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚552037546059264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43941327 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚554 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚546017258494446 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚537931195304720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚621490027856384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8461872291328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175185671 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚552037546059264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8461671857664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8116052115968 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123182828 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚546018046182218 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138217086 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138217088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175208687 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚560153616178176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43940773 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚613028131624960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚537931212344080 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138217088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 _step‚647 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175208687 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313402757 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚560152825530368 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123183384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79242057 _runtime‚8432.682883139 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8115279471104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚560152825530368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313425775 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138217086 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚560153616178176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚552037546059264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43941327 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚554 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚546017258494446 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚537931195304720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚621490027856384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8461872291328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175185671 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚552037546059264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8461671857664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8116052115968 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123182828 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚546018046182218 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138217086 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚8433.571606551 _step‚648 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175230424 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79251705 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8087834978224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚552105245998592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 % _timestamp‚1.7884963339290674e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175207382 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚552105264001536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚621566197008896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123197721 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚613103503907840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313441342 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138233962 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚546 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43946018 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537997278779840 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚621566417175040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175230424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138233960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8462717041152 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚537997261740480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123198269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43946564 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚546085113758064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚552105264001536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79251703 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚560221530215424 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175207382 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚552105245998592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313464386 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚560222320876544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8116284216832 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7899 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313464386 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125më"l. ó!…N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313441342 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8117056875008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8116284216832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7899 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313464386 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚546084326070084 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138233962 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138233960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚8433.571606551 _step‚648 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175230424 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79251705 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8087834978224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚552105245998592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 [MDeviceStatsMonitor.on_train_batch_end/i=çÏ×Ýnactive_split_bytes.small_pool.current‚ 196226048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 % _timestamp‚1.7884963339290674e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175207382 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚552105264001536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚621566197008896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123197721 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚613103503907840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313441342 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138233962 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚546 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43946018 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537997278779840 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚621566417175040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175230424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138233960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8462717041152 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚537997261740480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123198269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43946564 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚546085113758064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚552105264001536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79251703 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚560221530215424 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175207382 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚552105245998592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313464386 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚560222320876544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8116284216832 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8514377020928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139060786 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚563588074196992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8514176587264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 $ _timestamp‚1.788496378880287e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚555422543025664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139060788 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139060788 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚549371438555326 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176297589 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8136904423246 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚555422561028608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚625309725351424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123932861 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79724457 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚541234534132080 trainer/global_step‚7949 _step‚651 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315335359 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176274573 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8514377020928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139060786 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚563588074196992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8514176587264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 $ _timestamp‚1.788496378880287e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚555422543025664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139060788 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139060788 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚549371438555326 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176297589 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8136904423246 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚555422561028608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚625309725351424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123932861 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79724457 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚541234534132080 trainer/global_step‚7949 _step‚651 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315335359 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176274573 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8515423275520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8515231243776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176296284 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139077660 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚549437718443192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8166535917056 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _step‚652 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚563657569543168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚555 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚555490242964992 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44213677 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚616870897106944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚555490260967936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315396988 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123947225 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚555490242964992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123947782 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139077662 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚541300583528480 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚563656778882048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚8479.716964012 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8167308575232 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8166535917056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315396988 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315373944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 trainer/global_step‚7949 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚557 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44213122 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312¯½ e '…"…N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚625386128350720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176319326 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176296284 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139077660 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚549437718443192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8166535917056 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _step‚652 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚563657569543168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚555 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚555490242964992 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44213677 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚616870897106944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚555490260967936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315396988 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123947225 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚555490242964992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123947782 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139077662 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚541300583528480 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚563656778882048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚8479.716964012 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8167308575232 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8166535917056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315396988 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315373944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 trainer/global_step‚7949 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚557 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44213122 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8515423275520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8515231243776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80206855 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚552724043240662 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚558807557995008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8215782871552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8186204359942 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317267961 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚567023322863616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8567023779328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 _runtime‚8527.362915763 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚567023322863616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80206855 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚552724043240662 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚558807557995008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8215782871552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8186204359942 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317267961 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚567023322863616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8567023779328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 _runtime‚8527.362915763 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚567023322863616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44480776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚629206426026496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620638338186240 % _timestamp‚1.7884964286184182e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139921362 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183643136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚558875239931392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80216505 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚567092818209792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207583232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚629206218443264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8187976148440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177385186 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8216787617280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8217560275456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 317306546 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177408228 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139921362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80216503 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8187205499820 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚544603922355840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚552791110816300 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚558875239931392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8216787617280 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139921360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317329590 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8568087840256 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124696739 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚7999 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317329590 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124697281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44480236 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620638314246144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328å¬#—"…N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚558875257934336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚552791898504280 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 317306546 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44480776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚629206426026496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620638338186240 % _timestamp‚1.7884964286184182e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139921362 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183643136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚558875239931392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80216505 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚567092818209792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207583232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚629206218443264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8187976148440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177385186 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8216787617280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8217560275456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 317306546 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177408228 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139921362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80216503 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8187205499820 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚544603922355840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚552791110816300 UEDe* 8èviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚558875239931392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8216787617280 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139921360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317329590 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8568087840256 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124696739 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚7999 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317329590 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124697281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44480236 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620638314246144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176Vêž2˜"- _wandbruntime‚8528  _runtime‚8528阺\™"Ö 0loss/FRt2/assignment_loss‚3.0993368155077885 2loss/SRqqt2/assignment_loss‚1.3337739534952022 /loss/FRt3/detection_loss‚2.0837681389517253 .loss/FRt4/detection_loss‚1.835656167437824 1loss/SRqqt4/detection_loss‚1.0285576929019977 /loss/FBt3/detection_loss‚2.1310347574097768 %loss/total_loss‚41.33174538297635 epoch‚6 0loss/FRt4/assignment_loss‚1.3355781195255427 _step‚657 % _timestamp‚1.7884964286207714e+09 2loss/SRqqt4/assignment_loss‚0.8740885049882757 0loss/FBt1/assignment_loss‚0.7832173996455822 1loss/FBt2/assignment_loss‚0.12444919454156077 0loss/FRt3/assignment_loss‚2.2247863994704353 2loss/SRqqt3/assignment_loss‚0.8875219008392271 /loss/FBt1/detection_loss‚3.6835424661636353 _runtime‚8528.260943393 1loss/FBt4/assignment_loss‚0.12916009689840113 0loss/SRqqt1/detection_loss‚2.560723978739518 0loss/SRqqt2/detection_loss‚2.787808569493117 .loss/FBt2/detection_loss‚2.811001547745296 /loss/FBt4/detection_loss‚0.5039461102911874 trainer/global_step‚7999 2loss/SRqqt1/assignment_loss‚1.2113880592469985 .loss/FRt1/detection_loss‚2.347048451339728 /loss/FRt2/detection_loss‚2.4980737271088116 1loss/SRqqt3/detection_loss‚1.2526265317026306 /loss/FRt1/assignment_loss‚3.579312733913723 1loss/FBt3/assignment_loss‚0.22534406561836867/ëaš"Û 0loss/FRt2/assignment_loss‚3.0993368155077885 2loss/SRqqt2/assignment_loss‚1.3337739534952022 /loss/FRt3/detection_loss‚2.0837681389517253 .loss/FRt4/detection_loss‚1.835656167437824 1loss/SRqqt4/detection_loss‚1.0285576929019977 /loss/FBt3/detection_loss‚2.1310347574097768 %loss/total_loss‚41.33174538297635 epoch‚6 0loss/FRt4/assignment_loss‚1.3355781195255427 _step‚657 % _timestamp‚1.7884964286207714e+09 2loss/SRqqt4/assignment_loss‚0.8740885049882757 0loss/FBt1/assignment_loss‚0.7832173996455822 1loss/FBt2/assignment_loss‚0.12444919454156077 0loss/FRt3/assignment_loss‚2.2247863994704353 2loss/SRqqt3/assignment_loss‚0.8875219008392271 /loss/FBt1/detection_loss‚3.6835424661636353 _runtime‚8528.260943393 1loss/FBt4/assignment_loss‚0.12916009689840113 0loss/SRqqt1/detection_loss‚2.560723978739518 0loss/SRqqt2/detection_loss‚2.787808569493117 .loss/FBt2/detection_loss‚2.811001547745296 /loss/FBt4/detection_loss‚0.5039461102911874 trainer/global_step‚7999 2loss/SRqqt1/assignment_loss‚1.2113880592469985 .loss/FRt1/detection_loss‚2.347048451339728 /loss/FRt2/detection_loss‚2.4980737271088116 1loss/SRqqt3/detection_loss‚1.2526265317026306 /loss/FRt1/assignment_loss‚3.579312733913723 1loss/FBt3/assignment_loss‚0.22534406561836867‘·‹¸7’›"‚(:‡ ³”éԛ؛­ gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.436' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ·¾Õ‰œ"‚(:þ ³”éÔŠ¼™¯& disk.nvme0n1.out‚203865.6845703125 disk.dm-0.in‚28358.33203125 disk.sda3.out‚ 580.30078125 disk.sda.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1173.7576866149902# disk.dm-0.out‚203865.6845703125 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚93.16608905792236# proc.memory.rssMB‚ 2949.58203125 network.sent‚ 2.0086924e+07$ disk.nvme0c0n1.in‚28358.33203125" disk.nvme0n1.in‚28358.33203125 disk.drbd1260.out‚ 1.20703125 disk.sda2.out‚0 cpu‚25.282530739358293 network.recv‚ 2.839424e+06 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 580.30078125 disk.sda.out‚ 580.30078125 disk.sda1.out‚0) proc.memory.availableMB‚ 1119.66796875* proc.memory.percent‚18.002820014953613) disk./.usagePercent‚78.77186319702865( disk.nvme0c0n1.out‚203865.6845703125bÁ¤"‚(:… ”éÔÃÑ–­ gpu.0.gpu‚2.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.945 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ í¾ˆž"‚(:ý ”éÔ󟉰 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.2109375 disk.sda.in‚0) disk.nvme0c0n1.out‚204181.15673828125' disk.nvme0n1.out‚204181.15673828125 disk.sda.out‚ 580.64453125 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚93.1645393371582# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613) disk./.usagePercent‚78.77343559530394 disk.sda2.in‚0$ disk.nvme0c0n1.in‚28358.33203125" disk.nvme0n1.in‚28358.33203125$ disk.dm-0.out‚204181.15673828125 disk.dm-1.in‚0 cpu‚25.282237221284337 proc.cpu.threads‚17% disk./.usageGB‚1173.7811164855957 disk.dm-0.in‚28358.33203125 disk.sda3.in‚0' proc.memory.availableMB‚ 1119.921875 network.sent‚ 2.0194617e+07 network.recv‚ 2.844957e+06 disk.sda2.out‚0 disk.sda3.out‚ 580.64453125 disk.dm-1.out‚ 580.64453125<›’Ÿ"‚(:‡ Ñ”éÔýÞð¯ gpu.0.gpu‚22.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.583' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uߘy "‚(:î Ñ”éÔËñç² disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2949.58203125 network.recv‚ 2.849739e+06# disk.nvme0c0n1.in‚ 28528.0234375) disk.nvme0c0n1.out‚204306.94580078125! disk.nvme0n1.in‚ 28528.0234375 disk.dm-0.in‚ 28528.0234375 disk.sda.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1119.921875 network.sent‚ 2.0202785e+07) disk./.usagePercent‚78.76226829065286 disk.drbd1260.out‚ 1.2109375 disk.sda.out‚581.1875 disk.sda2.in‚0* proc.memory.percent‚18.002820014953613 proc.cpu.threads‚17$ disk.dm-0.out‚204306.94580078125 disk.dm-1.in‚0 disk.dm-1.out‚581.1875# memory_percent‚93.1645393371582 cpu‚25.282020129119754% disk./.usageGB‚1173.6147155761719' disk.nvme0n1.out‚204306.94580078125 disk.sda3.in‚0 disk.sda3.out‚581.1875MÔäV2¡"- _wandbruntime‚8577  _runtime‚8577r 4È¢" ' lr-AdamW/pg1‚0.00024038148082338243 ' lr-AdamW/pg2‚0.00024038148082338243 trainer/global_step‚8049 $ _timestamp‚1.788496478018187e+09 _runtime‚8577.658517504 _step‚658•­MÍ£"Ç $ _timestamp‚1.788496478018187e+09 _runtime‚8577.658517504 _step‚658 ' lr-AdamW/pg1‚0.00024038148082338243 ' lr-AdamW/pg2‚0.00024038148082338243 trainer/global_step‚8049’ÑMO‰2¤"- _wandbruntime‚8577  _runtime‚8577²M8ü'¥"öO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚632949637337088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8266034571776 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 trainer/global_step‚8049 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚556078223301542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884964780191762e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125432713 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140748188 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 319200563 _runtime‚8577.659675286 _step‚659 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140748186 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8266034571776 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140748186 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8266807216640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚562192554961408 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚556077435613770 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44742904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140748188 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚632949855419392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚547841160668720 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚570458571530240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178475393 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125432159 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚624330406982656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178452377 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8619254294528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚562192536958464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8619448436736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178452377 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80689257 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚570459362178048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8236274945050 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178475393 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005ûØe†(¦"ûO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140748186 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8266807216640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚562192554961408 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚556077435613770 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44742904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140748188 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚632949855419392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚547841160668720 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚570458571530240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178475393 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125432159 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚624330406982656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178452377 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8619254294528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚562192536958464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8619448436736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178452377 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80689257 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚570459362178048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8236274945050 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178475393 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚632949637337088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8266034571776 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 trainer/global_step‚8049 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚556078223301542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884964780191762e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125432713 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140748188 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 319200563 _runtime‚8577.659675286 _step‚659 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140748186 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8266034571776“Wö/2§"- _wandbruntime‚8578  _runtime‚8578ñV±›E¨"€N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.cÕõ³šÁ&urrent‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319262192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140765060 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8620510902272 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140765062 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178497130 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80698903 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183643136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 319262192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125447176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚570528066876416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125447736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8267811975680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8267811975680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚562260254900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚556144503189408 _runtime‚8578.501794007 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178474088 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8267039317504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 319239148 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 _step‚660 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140765060 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178497130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚562260236897792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80698905 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8620327259136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚8049 % _timestamp‚1.7884964788606224e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚562260254900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664ß,²¦ '©"…N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 319239148 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 _step‚660 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140765060 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178497130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚562260236897792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80698905 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8620327259136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚8049 % _timestamp‚1.7884964788606224e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚562260254900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319262192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140765060 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8620510902272 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140765062 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178497130 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80698903 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183643136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 319262192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125447176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚570528066876416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125447736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8267811975680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8267811975680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚562260254900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚556144503189408 _runtime‚8578.501794007 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178474088 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8267039317504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516”—£ºQ2ª"- _wandbruntime‚8578  _runtime‚8578„ z\«"Ö /loss/FRt1/detection_loss‚2.3911199597641826 % _timestamp‚1.7884964788633385e+09 /loss/FRt3/assignment_loss‚2.350914873506712 1loss/SRqqt2/assignment_loss‚1.386351338820532 2loss/SRqqt3/assignment_loss‚1.3108896295796486 0loss/FRt4/assignment_loss‚1.3000434706377428 /loss/FRt2/detection_loss‚2.0924752457071527 1loss/SRqqt1/detection_loss‚2.2452521473169327 epoch‚6 _runtime‚8578.503577888 1loss/SRqqt4/assignment_loss‚0.774044475518167 1loss/FBt2/assignment_loss‚0.09944323755335063 /loss/FRt4/detection_loss‚1.8073836524624292 1loss/SRqqt2/detection_loss‚2.1298266919329762 1loss/SRqqt4/detection_loss‚1.1702504666844527 .loss/FBt2/detection_loss‚2.377367651462555 0loss/FRt1/assignment_loss‚3.2341033152558585 1loss/FBt3/assignment_loss‚0.11206275430740789 /loss/FBt4/detection_loss‚0.6019312081348506 _step‚661 /loss/FRt2/assignment_loss‚3.278232499531337 0loss/FBt1/assignment_loss‚0.4194781241705641 /loss/FBt1/detection_loss‚2.2962338507175444 trainer/global_step‚8049 1loss/SRqqt1/assignment_loss‚1.309643054753542 1loss/FBt4/assignment_loss‚0.12243094843373792 /loss/FRt3/detection_loss‚2.1684577572766854 1loss/SRqqt3/detection_loss‚1.3788815331307873 /loss/FBt3/detection_loss‚2.3120667159557344 %loss/total_loss‚38.66888460261489¯Hœ a¬"Û epoch‚6 _runtime‚8578.503577888 1loss/SRqqt4/assignment_loss‚0.774044475518167 1loss/FBt2/assignment_loss‚0.09944323755335063 /loss/FRt4/detection_loss‚1.8073836524624292 1loss/SRqqt2/detection_loss‚2.1298266919329762 1loss/SRqqt4/detection_loss‚1.1702504666844527 .loss/FBt2/detection_loss‚2.377367651462555 0loss/FRt1/assignment_loss‚3.2341033152558585 1loss/FBt3/assignment_loss‚0.11206275430740789 /loss/FBt4/detection_loss‚0.6019312081348506 _step‚661 /loss/FRt2/assignment_loss‚3.278232499531337 0loss/FBt1/assignment_loss‚0.4194781241705641 /loss/FBt1/detection_loss‚2.2962338507175444 trainer/global_step‚8049 1loss/SRqqt1/assignment_loss‚1.309643054753542 1loss/FBt4/assignment_loss‚0.12243094843373792 /loss/FRt3/detection_loss‚2.1684577572766854 1loss/SRqqt3/detection_loss‚1.3788815331307873 /loss/FBt3/detection_loss‚2.3120667159557344 %loss/total_loss‚38.66888460261489 /loss/FRt1/detection_loss‚2.3911199597641826 % _timestamp‚1.7884964788633385e+09 /loss/FRt3/assignment_loss‚2.350914873506712 1loss/SRqqt2/assignment_loss‚1.386351338820532 2loss/SRqqt3/assignment_loss‚1.3108896295796486 0loss/FRt4/assignment_loss‚1.3000434706377428 /loss/FRt2/detection_loss‚2.0924752457071527 1loss/SRqqt1/detection_loss‚2.2452521473169327•i‹øþ’­"‚(:‡ à”éÔùÈ® gpu.0.gpu‚37.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.041' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…™Šiz®"‚(:ï à”éԙ̜± disk.sda1.out‚0# disk.nvme0c0n1.in‚ 28528.0234375$ proc.memory.availableMB‚1119.625$ disk.dm-0.out‚204466.54833984375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚204466.54833984375# proc.memory.rssMB‚ 2949.58203125 network.sent‚ 2.0206658e+07' disk.nvme0n1.out‚204466.54833984375 disk.dm-1.out‚ 581.53125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.21484375 disk.sda.in‚0 disk.sda2.out‚0* proc.memory.percent‚18.002820014953613 network.recv‚ 2.853714e+06$ disk./.usageGB‚1173.613052368164! disk.nvme0n1.in‚ 28528.0234375 disk.sda3.in‚0 disk.sda3.out‚ 581.53125$ memory_percent‚93.16635131835938 cpu‚25.281685489163834 proc.cpu.threads‚17) disk./.usagePercent‚78.76215667136792 disk.dm-0.in‚ 28528.0234375 disk.sda.out‚ 581.53125…Ó4œ¯"‚(:‘ ï”éÔ—믭 gpu.0.gpu‚24.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.72' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.544000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä—PC‰°"‚(:þ ï”éÔÜšÕ° disk.drbd1260.in‚0 disk.sda1.in‚0$ memory_percent‚93.16627979278564) proc.memory.availableMB‚ 1119.63671875 disk.dm-1.in‚0 disk.dm-1.out‚ 582.01171875 disk.drbd1260.out‚ 1.21484375# disk.nvme0c0n1.in‚ 28528.0234375! disk.nvme0n1.in‚ 28528.0234375' disk.nvme0n1.out‚204848.59912109375$ disk.dm-0.out‚204848.59912109375 network.sent‚ 2.0313751e+07% disk./.usageGB‚1172.9390563964844 disk.sda.in‚0 disk.sda.out‚ 582.01171875 disk.sda1.out‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚204848.59912109375 disk.dm-0.in‚ 28528.0234375 disk.sda2.out‚0# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613 cpu‚25.281345913212398 proc.cpu.threads‚17 network.recv‚ 2.859446e+06) disk./.usagePercent‚78.71692423618822 disk.sda3.in‚0 disk.sda3.out‚ 582.01171875“½=H±"‚(:’ þ”éÔꌰ® gpu.0.gpu‚30.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.959' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.791800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íœô…²"‚(:ú þ”éÔì¿°# proc.memory.rssMB‚ 2949.58203125 disk.drbd1260.out‚1.21875 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 28528.0234375) disk.nvme0c0n1.out‚205346.45556640625! disk.nvme0n1.in‚ 28528.0234375 disk.sda3.out‚ 582.69140625) proc.memory.availableMB‚ 1119.88671875 cpu‚25.281033881780683) disk./.usagePercent‚78.73068540917623 disk.drbd1260.in‚0' disk.nvme0n1.out‚205346.45556640625 disk.dm-0.in‚ 28528.0234375$ disk.dm-0.out‚205346.45556640625 disk.dm-1.in‚0 disk.dm-1.out‚ 582.69140625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0* proc.memory.percent‚18.002820014953613 proc.cpu.threads‚17 network.sent‚ 2.0320763e+07 network.recv‚ 2.862927e+06% disk./.usageGB‚1173.1441078186035 disk.sda.out‚ 582.69140625 disk.sda1.out‚0# memory_percent‚93.1647539138794—ç‰Î³"‚(:’ •éÔª® gpu.0.gpu‚33.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚136.998' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.399599999999996 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ö…´"‚(:ú •éÔ¶»„²* proc.memory.percent‚18.002820014953613 network.recv‚ 2.867403e+06) disk./.usagePercent‚78.73226036752688% disk./.usageGB‚1173.1675758361816 disk.sda.in‚0! disk.nvme0n1.in‚ 28704.8359375# memory_percent‚93.1647539138794 cpu‚25.280712822946736 proc.cpu.threads‚17 disk.sda3.out‚ 583.16015625 disk.drbd1260.out‚1.21875# disk.nvme0c0n1.in‚ 28704.8359375) disk.nvme0c0n1.out‚205634.02587890625 disk.dm-0.in‚ 28704.8359375) proc.memory.availableMB‚ 1119.88671875 network.sent‚ 2.0325149e+07 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 583.16015625 disk.sda2.in‚0$ disk.dm-0.out‚205634.02587890625 disk.dm-1.out‚ 583.16015625 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0' disk.nvme0n1.out‚205634.02587890625# proc.memory.rssMB‚ 2949.58203125v³â¹2µ"- _wandbruntime‚8628  _runtime‚8628„ä‘GǶ"Á trainer/global_step‚8099 % _timestamp‚1.7884965291088512e+09 _runtime‚8628.749350334 _step‚662 & lr-AdamW/pg1‚0.0002351399435375934 & lr-AdamW/pg2‚0.0002351399435375934_!AÌ·"Æ & lr-AdamW/pg1‚0.0002351399435375934 & lr-AdamW/pg2‚0.0002351399435375934 trainer/global_step‚8099 % _timestamp‚1.7884965291088512e+09 _runtime‚8628.749350334 _step‚662–ôG®Ç2¸"- _wandbruntime‚8628  _runtime‚8628Ëž‚H[¹"õO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573894610844672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8317058916864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81171655 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8672060566528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45011587 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179564295 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45011033 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚636769884688384 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚636769668703232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚565577533924864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8316286272000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141591886 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚628097824121856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚565577551927808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚559430827986878 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573893820196864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141591888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.free*ZÜ d‚628097800181760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚8628.750665652 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8317058916864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8316286272000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 trainer/global_step‚8099 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179564295 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179541279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141591886 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 $ _timestamp‚1.788496529110109e+09 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚551144499496080 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573894610844672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321133165 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81171657 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141591888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚559431615674650 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚551144482456720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8286345530158 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚554 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126183244 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573893820196864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179541279 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573893820196864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179541279 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45011587 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179564295 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45011033 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚636769884688384 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚636769668703232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚565577533924864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8316286272000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141591886 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚628097824121856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚565577551927808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚559430827986878 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573893820196864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141591888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚628097800181760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚8628.750665652 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8317058916864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8316286272000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 trainer/global_step‚8099 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179564295 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179541279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141591886 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 $ _timestamp‚1.788496529110109e+09 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚551144499496080 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573894610844672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321133165 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81171657 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141591888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚559431615674650 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚551144482456720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8286345530158 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚554 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126183244 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489—P;WŸ2»"- _wandbruntime‚8629  _runtime‚86296Ò'¼"€N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚565645251867136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚556 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚565645233864192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚636846070877184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141608760 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321194794 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8317291017728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573963315543040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45016302 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚636846286849024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚8099 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573963315543040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8287346670036 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126197605 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141608762 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8318063675904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81181305 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8672922352640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179586032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚628173148524544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179562990 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚559498683250496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚8629.712470041 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 321194794 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573962524881920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126198161 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8673114384384 % _timestamp‚1.7884965300708275e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚551210565931840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179586032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8317291017728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚551210548892480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚559497895562516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _step‚664 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126197605 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141608762 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8318063675904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81181305 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8672922352640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179586032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚628173148524544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179562990 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚559498683250496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚8629.712470041 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 321194794 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573962524881920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126198161 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8673114384384 % _timestamp‚1.7884965300708275e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚551210565931840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179586032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8317291017728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚551210548892480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚559497895562516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 _step‚664 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321194794 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8317291017728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1˜Ì¢ü@2¾"- _wandbruntime‚8629  _runtime‚8629û „Œ\¿"Ö 2loss/SRqqt4/assignment_loss‚0.8348526246641995 .loss/FBt1/detection_loss‚2.632291483879089 0loss/FBt1/assignment_loss‚0.3888942381599918 /loss/FBt4/detection_loss‚0.4657708363590279 _runtime‚8629.714574626 1loss/SRqqt1/assignment_loss‚1.462860211942877 /loss/FRt2/detection_loss‚2.2638807233984153 0loss/SRqqt4/detection_loss‚0.878445453895256 /loss/FBt2/detection_loss‚2.5772907654444377 &loss/total_loss‚41.958218639806006 /loss/FRt1/assignment_loss‚3.051925243215358 0loss/FRt2/assignment_loss‚3.1661092564463615 2loss/SRqqt2/assignment_loss‚1.6714947511968405 1loss/FBt2/assignment_loss‚0.20630488655283444 0loss/FBt4/assignment_loss‚0.1119854061769204 /loss/FRt1/detection_loss‚2.2288686469831367 /loss/FRt4/detection_loss‚1.4795923684501933 .loss/FBt3/detection_loss‚3.455981358885765 0loss/FRt4/assignment_loss‚1.5547668914946298 -loss/FRt3/detection_loss‚2.29216307225196 1loss/SRqqt1/detection_loss‚2.0896764499800544 % _timestamp‚1.7884965300743372e+09 _step‚665 0loss/FRt3/assignment_loss‚2.9963539728992865 1loss/SRqqt3/detection_loss‚1.8905033597155758 epoch‚6 trainer/global_step‚8099 2loss/SRqqt3/assignment_loss‚1.3885284352683178 3loss/FBt3/assignment_loss‚0.0028020435202051885 0loss/SRqqt2/detection_loss‚2.866876159025275L´’âaÀ"Û 2loss/SRqqt4/assignment_loss‚0.8348526246641995 .loss/FBt1/detection_loss‚2.632291483879089 0loss/FBt1/assignment_loss‚0.3888942381599918 /loss/FBt4/detection_loss‚0.4657708363590279 _runtime‚8629.714574626 1loss/SRqqt1/assignment_loss‚1.462860211942877 /loss/FRt2/detection_loss‚2.2638807233984153 0loss/SRqqt4/detection_loss‚0.878445453895256 /loss/FBt2/detection_loss‚2.5772907654444377 &loss/total_loss‚41.958218639806006 /loss/FRt1/assignment_loss‚3.051925243215358 0loss/FRt2/assignment_loss‚3.1661092564463615 2loss/SRqqt2/assignment_loss‚1.6714947511968405 1loss/FBt2/assignment_loss‚0.20630488655283444 0loss/FBt4/assignment_loss‚0.1119854061769204 /loss/FRt1/detection_loss‚2.2288686469831367 /loss/FRt4/detection_loss‚1.4795923684501933 .loss/FBt3/detection_loss‚3.455981358885765 0loss/FRt4/assignment_loss‚1.5547668914946298 -loss/FRt3/detection_loss‚2.29216307225196 1loss/SRqqt1/detection_loss‚2.0896764499800544 % _timestamp‚1.7884965300743372e+09 _step‚665 0loss/FRt3/assignment_loss‚2.9963539728992865 1loss/SRqqt3/detection_loss‚1.8905033597155758 epoch‚6 trainer/global_step‚8099 2loss/SRqqt3/assignment_loss‚1.3885284352683178 3loss/FBt3/assignment_loss‚0.0028020435202051885 0loss/SRqqt2/detection_loss‚2.866876159025275™Âàû’Á"‚(:‡ œ•éÔ¿¡Þ® gpu.0.gpu‚38.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.251' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w°kuÂ"‚(:ê œ•éÔ®Õµ°% disk./.usageGB‚1173.1659355163574 disk.dm-1.out‚ 583.609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 28704.8359375 disk.sda3.out‚ 583.609375$ memory_percent‚93.16668510437012( proc.memory.availableMB‚ 1119.5703125) disk./.usagePercent‚78.73215028428714 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.2265625 disk.sda.in‚0 disk.sda.out‚ 583.609375# disk.nvme0c0n1.in‚ 28704.8359375# disk.nvme0n1.out‚206018.1953125 network.recv‚ 2.871811e+06 disk.sda2.out‚0% disk.nvme0c0n1.out‚206018.1953125! disk.nvme0n1.in‚ 28704.8359375 disk.dm-0.out‚206018.1953125 disk.sda3.in‚0# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613 network.sent‚ 2.0330917e+07 disk.sda2.in‚0 cpu‚25.280382881715887 proc.cpu.threads‚17¢TXãÃ"‚(:’ «•éÔÎÇá® gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.872' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.574400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 3QvÄ"‚(:ë «•éԉž³ disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0n1.out‚ 206121.640625 disk.dm-0.out‚ 206121.640625 disk.sda3.in‚0# proc.memory.rssMB‚ 2949.58203125 cpu‚25.28002772412625 network.sent‚ 2.0438714e+07# disk.nvme0c0n1.in‚ 28704.8359375 disk.dm-0.in‚ 28704.8359375 disk.sda3.out‚ 583.94921875$ memory_percent‚93.16740036010742* proc.memory.percent‚18.002820014953613 network.recv‚ 2.877437e+06) disk./.usagePercent‚78.73267842783028 disk.sda2.in‚0$ disk.nvme0c0n1.out‚ 206121.640625 disk.dm-1.out‚ 583.94921875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.2265625' proc.memory.availableMB‚ 1119.453125% disk./.usageGB‚1173.1738052368164 disk.sda.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 28704.8359375 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 583.94921875ÝÜh’Å"‚(:‡ º•éÔ•Þ® gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚127.607' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j›~Æ"‚(:ö º•éÔûÑÚ²% disk./.usageGB‚1173.1738052368164 disk.sda3.out‚ 584.546875' proc.memory.availableMB‚ 1119.453125# proc.memory.rssMB‚ 2949.58203125 cpu‚25.279779335736478 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 2.0443886e+07 network.recv‚ 2.881038e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 584.546875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.23046875 disk.sda.out‚ 584.546875 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 28704.8359375' disk.nvme0n1.out‚206361.49267578125 disk.dm-0.in‚ 28704.8359375$ disk.dm-0.out‚206361.49267578125 disk.sda3.in‚0) disk.nvme0c0n1.out‚206361.49267578125! disk.nvme0n1.in‚ 28704.8359375$ memory_percent‚93.16740036010742* proc.memory.percent‚18.002820014953613) disk./.usagePercent‚78.732678427830280ìŠ2Ç"- _wandbruntime‚8677  _runtime‚8677¢VzáÉÈ"à ' lr-AdamW/pg1‚0.00022993195447396685 ' lr-AdamW/pg2‚0.00022993195447396685 trainer/global_step‚8149 % _timestamp‚1.7884965774087152e+09 _runtime‚8677.049038021 _step‚666èîAXÎÉ"È ' lr-AdamW/pg2‚0.00022993195447396685 trainer/global_step‚8149 % _timestamp‚1.7884965774087152e+09 _runtime‚8677.049038021 _step‚666 ' lr-AdamW/pg1‚0.00022993195447396685š²Æc2Ê"- _wandbruntime‚8677  _runtime‚8677ÒWØéü'Ë"öO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚577329068863488 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142435588 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180630181 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180653197 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚631865241261056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚554447804244720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚568962530891264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45280412 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚577329859511296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 trainer/global_step‚8149 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126933908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8337186763678 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 323065767 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8366537972224 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180630181 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚667 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚631865217320960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8367310617088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚562785008047758 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180653197 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚568962530891264 _runtime‚8677.051087181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚568962548894208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884965774099157e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8336416115266 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚577329068863488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142435586 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚554447821284080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚640590124757504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚562784220359986 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142435588 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8724883496448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8724685159936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8366537972224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 323088785 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126934469 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81654055 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚640589902480896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142435586 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180630181 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180653197 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚631865241261056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚554447804244720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚568962530891264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45280412 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚577329859511296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 trainer/global_step‚8149 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126933908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8337186763678 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 323065767 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8366537972224 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180630181 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚667 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚631865217320960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8367310617088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚562785008047758 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180653197 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚568962530891264 _runtime‚8677.051087181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚568962548894208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884965774099157e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8336416115266 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚577329068863488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142435586 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚554447821284080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚640590124757504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚562784220359986 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0›IÙ¬y2Í"- _wandbruntime‚8677  _runtime‚8677ëÿÂ6Î"ÿM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚577397773548544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8368315376128 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚577398564209664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 323104352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323127396 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180674934 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208808960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 122ó¡øÏ!155092992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚556 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142452462 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142452460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126949386 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81663705 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180674934 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 323104352 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126948828 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81663703 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚562852075623604 trainer/global_step‚8149 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8367542717952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8367542717952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8338187903764 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚569030248833536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚8677.987208304 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚569030230830592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142452460 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚569030230830592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180651892 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚640666284105216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 _step‚668 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8368315376128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 $ _timestamp‚1.788496578345815e+09 ‰ˆ 'Ï"„N PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81663703 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚562852075623604 trainer/global_step‚8149 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8367542717952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8367542717952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8338187903764 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚569030248833536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚8677.987208304 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚569030230830592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142452460 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚569030230830592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180651892 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚640666284105216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 _step‚668 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8368315376128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 $ _timestamp‚1.788496578345815e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚577397773548544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8368315376128 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚577398564209664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 323104352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323127396 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180674934 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208808960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚556 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142452462 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142452460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126949386 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81663705 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180674934 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 323104352 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126948828 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39œ¶ñ_Ÿ2Ð"- _wandbruntime‚8677  _runtime‚8677^iŒžZÑ"Ô /loss/FRt4/detection_loss‚1.5538234957430053 1loss/SRqqt2/detection_loss‚2.2844796061515806 1loss/SRqqt3/detection_loss‚1.6203050293812626 0loss/FRt2/assignment_loss‚2.8395868274900646 2loss/SRqqt3/assignment_loss‚1.4530006835090095 .loss/FBt1/detection_loss‚2.196096828350654 /loss/FBt3/detection_loss‚1.9707035660743712 _runtime‚8677.989764919 0loss/FBt3/assignment_loss‚0.0422112154225033 &loss/total_loss‚39.889907535877015 /loss/FBt4/detection_loss‚0.7106401617638767 epoch‚6 trainer/global_step‚8149 .loss/FRt1/assignment_loss‚3.69058579336042 0loss/FRt3/assignment_loss‚2.6796301667754716 2loss/SRqqt4/assignment_loss‚0.9854341110266331 /loss/FBt2/assignment_loss‚0.296129706451451 .loss/FRt1/detection_loss‚2.251588732652042 /loss/FRt4/assignment_loss‚1.190093408708703 1loss/SRqqt1/assignment_loss‚1.468853492205116 2loss/SRqqt2/assignment_loss‚1.6303524205727236 1loss/FBt4/assignment_loss‚0.06450032103771264 /loss/FBt2/detection_loss‚2.5516215761502585 0loss/SRqqt1/detection_loss‚2.580083256418055 _step‚669 1loss/FBt1/assignment_loss‚0.23367829738975843 /loss/FRt2/detection_loss‚2.3995426639198465 1loss/SRqqt4/detection_loss‚0.8250535636237173 % _timestamp‚1.7884965783490803e+09 /loss/FRt3/detection_loss‚2.3719126116987823Z5_Ò"Ù /loss/FRt2/detection_loss‚2.3995426639198465 1loss/SRqqt4/detection_loss‚0.8250535636237173 % _timestamp‚1.7884965783490803e+09 /loss/FRt3/detection_loss‚2.3719126116987823 /loss/FRt4/detection_loss‚1.5538234957430053 1loss/SRqqt2/detection_loss‚2.2844796061515806 1loss/SRqqt3/detection_loss‚1.6203050293812626 0loss/FRt2/assignment_loss‚2.8395868274900646 2loss/SRqqt3/assignment_loss‚1.4530006835090095 .loss/FBt1/detection_loss‚2.196096828350654 /loss/FBt3/detection_loss‚1.9707035660743712 _runtime‚8677.989764919 0loss/FBt3/assignment_loss‚0.0422112154225033 &loss/total_loss‚39.889907535877015 /loss/FBt4/detection_loss‚0.7106401617638767 epoch‚6 trainer/global_step‚8149 .loss/FRt1/assignment_loss‚3.69058579336042 0loss/FRt3/assignment_loss‚2.6796301667754716 2loss/SRqqt4/assignment_loss‚0.9854341110266331 /loss/FBt2/assignment_loss‚0.296129706451451 .loss/FRt1/detection_loss‚2.251588732652042 /loss/FRt4/assignment_loss‚1.190093408708703 1loss/SRqqt1/assignment_loss‚1.468853492205116 2loss/SRqqt2/assignment_loss‚1.6303524205727236 1loss/FBt4/assignment_loss‚0.06450032103771264 /loss/FBt2/detection_loss‚2.5516215761502585 0loss/SRqqt1/detection_loss‚2.580083256418055 _step‚669 1loss/FBt1/assignment_loss‚0.23367829738975843¿„Y&Ó"‚(:’ É•éÔª® gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚128.428' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.685599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ï_Õ„Ô"‚(:ù É•éÔ󽺮$ disk.dm-0.out‚206627.70361328125 disk.sda3.in‚0 disk.sda3.out‚ 584.98828125 disk.drbd1260.out‚ 1.23046875# memory_percent‚93.1675910949707# proc.memory.rssMB‚ 2949.58203125! disk.nvme0n1.in‚ 28868.1484375' disk.nvme0n1.out‚206627.70361328125 network.sent‚ 2.0450044e+07 network.recv‚ 2.885719e+06) disk./.usagePercent‚78.72551328896196 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 584.98828125# disk.nvme0c0n1.in‚ 28868.1484375 disk.dm-0.in‚ 28868.1484375 disk.drbd1260.in‚0' proc.memory.availableMB‚ 1119.421875 cpu‚25.27949717038011) disk.nvme0c0n1.out‚206627.70361328125 disk.sda.out‚ 584.98828125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚18.002820014953613 proc.cpu.threads‚17$ disk./.usageGB‚1173.067039489746 disk.sda.in‚0|âŸÕ"‚(:… Ø•éÔ¸Íø¯ gpu.0.gpu‚21.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.28' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.456 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ùí;¢‚Ö"‚(:÷ Ø•éÔÚ—¹± disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 585.41796875 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1119.41796875 network.recv‚ 2.892076e+06) disk./.usagePercent‚78.73978212489038 disk.drbd1260.out‚1.234375 disk.sda2.out‚0* proc.memory.percent‚18.002820014953613 proc.cpu.threads‚17 network.sent‚ 2.0558515e+07 disk.dm-0.in‚ 28868.1484375# disk.dm-0.out‚207091.6416015625 disk.sda.out‚ 585.41796875 disk.sda1.out‚0$ memory_percent‚93.16761493682861# proc.memory.rssMB‚ 2949.58203125$ disk./.usageGB‚1173.279655456543# disk.nvme0c0n1.in‚ 28868.1484375( disk.nvme0c0n1.out‚207091.6416015625& disk.nvme0n1.out‚207091.6416015625 disk.sda3.out‚ 585.41796875 disk.sda2.in‚0 cpu‚25.27914418271599! disk.nvme0n1.in‚ 28868.1484375k‘×"‚(:† ç•éÔ¦Ó¯® gpu.0.gpu‚27.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.935' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.587 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tò Ø"‚(:ö ç•éԜۃ²) disk./.usagePercent‚78.75408961795648# disk.nvme0c0n1.in‚ 28868.1484375 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 1119.4140625# proc.memory.rssMB‚ 2949.58203125 proc.cpu.threads‚17 network.recv‚ 2.895714e+06! disk.nvme0n1.in‚ 28868.1484375 disk.dm-0.in‚ 28868.1484375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚1.234375 disk.sda1.out‚0 disk.sda2.out‚0 network.sent‚ 2.0563634e+07( disk.nvme0c0n1.out‚207469.0048828125& disk.nvme0n1.out‚207469.0048828125 disk.dm-1.out‚ 585.65234375 disk.sda.out‚ 585.65234375$ memory_percent‚93.16763877868652* proc.memory.percent‚18.002820014953613 cpu‚25.27886366665956$ disk./.usageGB‚1173.492847442627# disk.dm-0.out‚207469.0048828125 disk.sda3.out‚ 585.65234375 disk.sda.in‚0,Î^W2Ù"- _wandbruntime‚8727  _runtime‚8727ù'y¾ÉÚ"à trainer/global_step‚8199 % _timestamp‚1.7884966274163144e+09 _runtime‚8727.056636396 _step‚670 ' lr-AdamW/pg1‚0.00022475859896961756 ' lr-AdamW/pg2‚0.00022475859896961756°B¸'ÎÛ"È ' lr-AdamW/pg2‚0.00022475859896961756 trainer/global_step‚8199 % _timestamp‚1.7884966274163144e+09 _runtime‚8727.056636396 _step‚670 ' lr-AdamW/pg1‚0.00022475859896961756ž°Wõˆ2Ü"- _wandbruntime‚8727  _runtime‚8727çr:ŒÝ"öO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 324998369 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181719083 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚635632634460160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127684300 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82136455 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8417562317312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8777493862400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚572347545860608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚557751143072080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 trainer/global_step‚8199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚644409938374656 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 % _timestamp‚1.7884966274173105e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚580764317530112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635632658400256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚566137612733094 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8416789672448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚557751126032720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82136457 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8417562317312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚580765108177920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45547281 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143279286 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181719083 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚566138400420866 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181742099 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _step‚671 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 325021387 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚557751143072080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 trainer/global_step‚8199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚644409938374656 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 % _timestamp‚1.7884966274173105e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚580764317530112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635632658400256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚566137612733094 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8416789672448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚557751126032720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82136457 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8417562317312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚580765108177920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45547281 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143279286 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181719083 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚566138400420866 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181742099 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _step‚671 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 325021387 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8727.915017088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325036954 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82146105 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127699283 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884966282738645e+09 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181740794 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚644486557644800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127698715 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143296160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚572415245799936 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181763836 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8388258488872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181740794 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚580833022215168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82146103 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325036954 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8417794418176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚566204680308732 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8387487840252 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325059998 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45552612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143296162 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚572415227796992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚644486337478656 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚557817209507840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚580833812876288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143296160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8778550901760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8417794418176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0³ß$4#á"…N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181763836 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45553178 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8418567076352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚566205467996712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _step‚672 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325059998 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚572415227796992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚635708006743040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚635707982802944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8778354675712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8727.915017088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325036954 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82146105 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127699283 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884966282738645e+09 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181740794 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚644486557644800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127698715 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143296160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚572415245799936 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181763836 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8388258488872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181740794 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚580833022215168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82146103 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325036954 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8417794418176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚566204680308732 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8387487840252 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325059998 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45552612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143296162 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚572415227796992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚644486337478656 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚557817209507840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚580833812876288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143296160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8778550901760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8417794418176 €¶‘Î2â"- _wandbruntime‚8727  _runtime‚8727Œv’ÎVã"Ð .loss/FRt3/detection_loss‚2.541445211360329 1loss/SRqqt4/detection_loss‚0.8013565230273432 %loss/total_loss‚42.81753185994682 % _timestamp‚1.7884966282763577e+09 /loss/FBt4/detection_loss‚0.5418262770266405 0loss/FRt1/assignment_loss‚3.3561963909550716 0loss/FRt4/assignment_loss‚1.1322103195023119 _step‚673 0loss/FRt3/assignment_loss‚2.2166116974855723 /loss/FRt4/detection_loss‚1.5263902451516242 0loss/SRqqt1/detection_loss‚2.786997005343437 /loss/FBt1/detection_loss‚2.4870388167245046 trainer/global_step‚8199 1loss/FBt4/assignment_loss‚0.26504542933718084 2loss/SRqqt3/assignment_loss‚1.2747462939471006 0loss/SRqqt4/assignment_loss‚0.83732695971185 0loss/FBt2/assignment_loss‚0.5961902244402495 _runtime‚8727.916537392 epoch‚6 /loss/SRqqt3/detection_loss‚1.88616074770689 .loss/FRt1/detection_loss‚2.461492749715322 .loss/FRt2/detection_loss‚2.947013552724918 0loss/SRqqt1/assignment_loss‚1.34698374910901 1loss/FBt1/assignment_loss‚0.14765255561437943 1loss/FBt3/assignment_loss‚0.07830803145423791 0loss/SRqqt2/detection_loss‚2.458651194649358 /loss/FRt2/assignment_loss‚3.273564342941557 2loss/SRqqt2/assignment_loss‚1.8573551524670855 /loss/FBt2/detection_loss‚3.8427996089061103 /loss/FBt3/detection_loss‚2.1541687806447345hšõ[ä"Õ 0loss/FRt3/assignment_loss‚2.2166116974855723 /loss/FRt4/detection_loss‚1.5263902451516242 0loss/SRqqt1/detection_loss‚2.786997005343437 /loss/FBt1/detection_loss‚2.4870388167245046 trainer/global_step‚8199 1loss/FBt4/assignment_loss‚0.26504542933718084 2loss/SRqqt3/assignment_loss‚1.2747462939471006 0loss/SRqqt4/assignment_loss‚0.83732695971185 0loss/FBt2/assignment_loss‚0.5961902244402495 _runtime‚8727.916537392 epoch‚6 /loss/SRqqt3/detection_loss‚1.88616074770689 .loss/FRt1/detection_loss‚2.461492749715322 .loss/FRt2/detection_loss‚2.947013552724918 0loss/SRqqt1/assignment_loss‚1.34698374910901 1loss/FBt1/assignment_loss‚0.14765255561437943 1loss/FBt3/assignment_loss‚0.07830803145423791 0loss/SRqqt2/detection_loss‚2.458651194649358 /loss/FRt2/assignment_loss‚3.273564342941557 2loss/SRqqt2/assignment_loss‚1.8573551524670855 /loss/FBt2/detection_loss‚3.8427996089061103 /loss/FBt3/detection_loss‚2.1541687806447345 .loss/FRt3/detection_loss‚2.541445211360329 1loss/SRqqt4/detection_loss‚0.8013565230273432 %loss/total_loss‚42.81753185994682 % _timestamp‚1.7884966282763577e+09 /loss/FBt4/detection_loss‚0.5418262770266405 0loss/FRt1/assignment_loss‚3.3561963909550716 0loss/FRt4/assignment_loss‚1.1322103195023119 _step‚673¡3¨vR’å"‚(:‡ ö•éÔ´Ž± gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð’;5‡æ"‚(:ü ö•éÔí°œ² disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 586.16015625 disk.drbd1260.out‚ 1.23828125 disk.sda1.out‚0# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613 cpu‚25.278614599806293 network.recv‚ 2.89925e+06) disk.nvme0c0n1.out‚207854.58642578125! disk.nvme0n1.in‚ 28868.1484375 disk.dm-1.in‚0 disk.sda.out‚ 586.16015625$ memory_percent‚93.16802024841309 proc.cpu.threads‚17 network.sent‚ 2.0568607e+07' disk.nvme0n1.out‚207854.58642578125$ disk.dm-0.out‚207854.58642578125 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1119.3515625 disk.dm-1.out‚ 586.16015625) disk./.usagePercent‚78.75405710499963% disk./.usageGB‚1173.4923629760742# disk.nvme0c0n1.in‚ 28868.1484375 disk.dm-0.in‚ 28868.1484375JXa‘ç"‚(:† …–éÔŽµ¿® gpu.0.gpu‚24.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.023 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€Ï_©ƒè"‚(:ø …–éÔ…²Ë±! disk.nvme0c0n1.in‚ 28930.59375 disk.sda3.in‚0 disk.drbd1260.out‚ 1.23828125* proc.memory.percent‚18.002820014953613 cpu‚25.27834314159067 network.sent‚ 2.0677412e+07 disk.sda.out‚ 586.70703125) proc.memory.availableMB‚ 1119.35546875# proc.memory.rssMB‚ 2949.58203125 proc.cpu.threads‚17 network.recv‚ 2.905803e+06 disk.sda2.out‚0' disk.nvme0n1.out‚208052.52392578125$ disk.dm-0.out‚208052.52392578125 disk.dm-1.out‚ 586.70703125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚78.77118349702523) disk.nvme0c0n1.out‚208052.52392578125 disk.nvme0n1.in‚ 28930.59375 disk.dm-0.in‚28930.98046875 disk.sda3.out‚ 586.70703125 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚93.16799640655518# disk./.usageGB‚1173.74755859375 disk.sda2.in‚0ñ¯‡K…é"‚(:ú ”–éÔ‚¢ù°# proc.memory.rssMB‚ 2949.58203125 cpu‚25.278100950579415% disk./.usageGB‚1173.7475509643555 disk.dm-0.in‚29008.53515625 disk.sda.in‚0 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.sda.out‚ 587.1015625 disk.sda1.out‚0$ disk.nvme0c0n1.in‚29008.53515625) disk.nvme0c0n1.out‚208268.25341796875$ memory_percent‚93.16959381103516& proc.memory.availableMB‚ 1119.09375* proc.memory.percent‚18.002820014953613 network.sent‚ 2.0683381e+07 network.recv‚ 2.910061e+06" disk.nvme0n1.in‚29008.53515625' disk.nvme0n1.out‚208268.25341796875 disk.dm-1.in‚0 disk.dm-1.out‚ 587.1015625 disk.drbd1260.out‚ 1.2421875 disk.sda2.in‚0) disk./.usagePercent‚78.77118298501017$ disk.dm-0.out‚208268.25341796875 disk.sda3.in‚0 disk.sda3.out‚ 587.1015625 disk.sda1.in‚0 disk.sda2.out‚0p¬ŒL‘ê"‚(:† ”–éÔ価± gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚122.935' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.587 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úr|ë"‚(:’ £–éÔÒïð± gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.416' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.083199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôóÈì"‚(:ý £–éÔ¨ßÁ´ proc.cpu.threads‚17 network.sent‚ 2.0688589e+07 disk.dm-1.out‚ 587.52734375$ disk.nvme0c0n1.in‚29008.53515625) disk.nvme0c0n1.out‚208797.30029296875' disk.nvme0n1.out‚208797.30029296875$ memory_percent‚93.16649436950684( proc.memory.availableMB‚ 1119.6015625# proc.memory.rssMB‚ 2949.58203125# disk./.usageGB‚1173.74755859375 disk.drbd1260.out‚ 1.2421875 disk.sda.in‚0 disk.sda.out‚ 587.52734375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚29008.53515625 network.recv‚ 2.913866e+06 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0$ disk.dm-0.out‚208797.30029296875* proc.memory.percent‚18.002820014953613 cpu‚25.277806094459013) disk./.usagePercent‚78.77118349702523 disk.sda3.out‚ 587.52734375" disk.nvme0n1.in‚29008.53515625iH…2í"- _wandbruntime‚8777  _runtime‚8777>ÇïÅÇî"Á % _timestamp‚1.7884966782073026e+09 _runtime‚8777.847650251 _step‚674 & lr-AdamW/pg1‚0.0002196209551440788 & lr-AdamW/pg2‚0.0002196209551440788 trainer/global_step‚8249Ña´¯Ìï"Æ _step‚674 & lr-AdamW/pg1‚0.0002196209551440788 & lr-AdamW/pg2‚0.0002196209551440788 trainer/global_step‚8249 % _timestamp‚1.7884966782073026e+09 _runtime‚8777.847650251¢úA»c2ð"- _wandbruntime‚8777  _runtime‚8777ô²gü'ñ"öO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182807985 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8830194866176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45815703 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚639400051599360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8436557285482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144122986 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128434560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 326930971 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584199566196736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326953989 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚639400075539456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8830009112576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144122986 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚575732542827008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584199566196736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144122988 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45815159 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8467041372672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8467814017536 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182807985 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚575732524824064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚561054447820720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82618855 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚8777.849102197 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚569491005106202 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8467041372672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144122988 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚648230060711936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚569491792793974 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8467814017536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚561054464860080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185753600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8437327933894 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 % _timestamp‚1.7884966782086155e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182831001 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 182831001 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82618857 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚584200356844544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚584200356844544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚648230270405632 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _step‚675 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209693696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚575732524824064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825us!>(ò"ûO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚561054464860080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185753600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8437327933894 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 % _timestamp‚1.7884966782086155e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182831001 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 182831001 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82618857 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚584200356844544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚584200356844544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚648230270405632 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _step‚675 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209693696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚575732524824064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182807985 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8830194866176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45815703 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚639400051599360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8436557285482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144122986 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128434560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 326930971 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584199566196736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326953989 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚639400075539456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8830009112576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144122986 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚575732542827008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584199566196736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144122988 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45815159 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8467041372672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8467814017536 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182807985 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚575732524824064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚561054447820720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82618855 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚8777.849102197 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚569491005106202 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8467041372672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144122988 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚648230060711936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚569491792793974 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8467814017536£>[Å2ó"- _wandbruntime‚8778  _runtime‚8778< Šd, ô"ÿM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚569558860369820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚676 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 182829696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326969556 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144139860 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128448975 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8831253885952 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚8778.801058046 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144139862 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326992600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326992600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182852738 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚554 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45821024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584269061542912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚561120531295840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005Ân°ò 'õ"„N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8831061854208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚575800242766336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8468046118400 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82628505 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45820472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚575800224763392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584268270881792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182829696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚561120514256480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8438329073980 $ _timestamp‚1.788496679159416e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8468818776576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8468046118400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚8778.801058046 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144139862 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326992600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326992600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182852738 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚554 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45821024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584269061542912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚561120531295840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚569558860369820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚676 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 182829696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326969556 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144139860 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128448975 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8831253885952 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 Fºç‘`ø"Ú /loss/FRt3/assignment_loss‚2.631321269732255 0loss/SRqqt2/detection_loss‚2.628116490840912 0loss/SRqqt3/detection_loss‚1.505351613335153 0loss/FRt4/assignment_loss‚1.3499019192714317 0loss/FBt1/assignment_loss‚0.7229065142571927 /loss/FRt2/detection_loss‚2.1397863406114417 _step‚677 2loss/SRqqt1/assignment_loss‚1.6978166437276252 2loss/SRqqt2/assignment_loss‚1.9189629542827606 2loss/SRqqt4/assignment_loss‚0.8836558347245977 2loss/FBt2/assignment_loss‚0.033130170342167274 0loss/SRqqt1/detection_loss‚2.100162250239675 .loss/FBt2/detection_loss‚2.495045837759972 .loss/FBt3/detection_loss‚2.888057385881742 /loss/FBt4/detection_loss‚0.6413616248925181 .loss/FRt1/assignment_loss‚3.02946218252182 3loss/FBt3/assignment_loss‚0.0006507664929813473 trainer/global_step‚8249 /loss/FRt4/detection_loss‚1.5815685189456916 1loss/SRqqt4/detection_loss‚0.7378989919836139 &loss/total_loss‚42.757713570532175 2loss/SRqqt3/assignment_loss‚1.5761977715932942 /loss/FBt1/detection_loss‚3.6459140539169312 _runtime‚ 8778.80243986 .loss/FRt2/assignment_loss‚3.70793961897129 1loss/FBt4/assignment_loss‚0.04052973906861633 /loss/FRt3/detection_loss‚2.8057258043151636 epoch‚6 /loss/FRt1/detection_loss‚1.9962492728233336 % _timestamp‚1.7884966791619353e+09¥Ý›Òù"‚(:’ ²–é԰Ⱦ¯ gpu.0.gpu‚20.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚165.837' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.167399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î Š‚ú"‚(:÷ ²–éÔº‚µ² disk.dm-1.in‚0 disk.drbd1260.in‚0 cpu‚25.27754674933839% disk./.usageGB‚1174.2886581420898( disk.nvme0c0n1.out‚209151.0048828125# disk.dm-0.out‚209151.0048828125 disk.sda3.out‚ 588.08984375$ memory_percent‚93.16823482513428* proc.memory.percent‚18.002820014953613 network.sent‚ 2.0693678e+07 network.recv‚ 2.917233e+06$ disk.nvme0c0n1.in‚29008.53515625" disk.nvme0n1.in‚29008.53515625& disk.nvme0n1.out‚209151.0048828125 disk.sda3.in‚0 disk.dm-1.out‚ 588.08984375 proc.cpu.threads‚17) disk./.usagePercent‚78.80749714172708 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.out‚1.25) proc.memory.availableMB‚ 1119.31640625# proc.memory.rssMB‚ 2949.58203125 disk.sda.out‚ 588.08984375 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚29008.53515625†ïÕû"‚(:… Á–éÔÂöDz gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.945 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ËÙc{ü"‚(:ð Á–éÔŽÕü¶ disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚29008.53515625( proc.memory.availableMB‚ 1119.3203125* proc.memory.percent‚18.002820014953613) disk./.usagePercent‚78.74053171495065 disk.dm-1.out‚ 588.578125 disk.drbd1260.out‚1.25 disk.sda.in‚0( disk.nvme0c0n1.out‚209759.2119140625# disk.dm-0.out‚209759.2119140625 network.sent‚ 2.0802467e+07 disk.sda.out‚ 588.578125" disk.nvme0n1.in‚29008.53515625# proc.memory.rssMB‚ 2949.58203125 cpu‚25.277304297887742 proc.cpu.threads‚17% disk./.usageGB‚1173.2908248901367 disk.sda3.in‚0 disk.sda3.out‚ 588.578125 disk.dm-1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚209759.2119140625 disk.dm-0.in‚29008.53515625$ memory_percent‚93.16821098327637 network.recv‚ 2.92397e+06 disk.drbd1260.in‚0 À í‘ý"‚(:† ЖéÔ¬¼¬® gpu.0.gpu‚22.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.023 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L'p^‚þ"‚(:÷ ЖéÔìí²( disk.nvme0c0n1.out‚209920.7802734375) proc.memory.availableMB‚ 1119.05859375* proc.memory.percent‚18.002820014953613 disk.sda.out‚ 589.078125 network.recv‚ 2.928442e+06) disk./.usagePercent‚78.74891672971962 disk.dm-1.out‚ 589.078125$ disk.nvme0c0n1.in‚29136.12890625" disk.nvme0n1.in‚29136.12890625& disk.nvme0n1.out‚209920.7802734375# proc.memory.rssMB‚ 2949.58203125% disk./.usageGB‚1173.4157676696777 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚25.27708574491236# disk.dm-0.out‚209920.7802734375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.25390625 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚93.16980838775635 proc.cpu.threads‚17 network.sent‚ 2.0810301e+07 disk.dm-0.in‚29136.12890625 disk.sda3.out‚ 589.078125 disk.drbd1260.in‚0 disk.sda1.in‚0Á¤ÿ2ÿ"- _wandbruntime‚8827  _runtime‚8827Ø7ÅàÉ€#à trainer/global_step‚8299 % _timestamp‚1.7884967279103932e+09 _runtime‚8827.550945228 _step‚678 ' lr-AdamW/pg1‚0.00021452009367462345 ' lr-AdamW/pg2‚0.00021452009367462345bãÕêÎ#È ' lr-AdamW/pg1‚0.00021452009367462345 ' lr-AdamW/pg2‚0.00021452009367462345 trainer/global_step‚8299 % _timestamp‚1.7884967279103932e+09 _runtime‚8827.550945228 _step‚678¦­À„Á2‚#- _wandbruntime‚8827  _runtime‚8827{ƒˆƒ#öO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8882912275456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144966688 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46083814 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚564357769608720 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129185071 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8518065717760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8518065717760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83101255 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚579117521790464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚587635605511168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚579117539793408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 % _timestamp‚1.7884967279112844e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚643167468738560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚579117521790464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8517293072896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226470912 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144966686 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚572845185167082 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚572844397479310 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚587634814863360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83101257 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202530816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚587634814863360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8487398519002 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚8299 _step‚679 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144966688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183896887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46083264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183919903 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚579117539793408 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183919903 _runtime‚8827.551808094 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚652050178483200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 N@DeviceStatsMonitor.on_train_byØÀtatch_start/allocation.all.allocated‚ 328886591 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144966686 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚572845185167082 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚572844397479310 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚587634814863360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83101257 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202530816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚587634814863360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8487398519002 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚8299 _step‚679 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144966688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183896887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46083264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183919903 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚579117539793408 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183919903 _runtime‚8827.551808094 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚652050178483200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 328886591 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚579117521790464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚587635605511168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚579117539793408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489§ælÎ72…#- _wandbruntime‚8828  _runtime‚8828xéwÚ'†#€N [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚579185239732736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144983560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8518297818624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚572912252742928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129199519 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46089151 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144983560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8883948722176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8488399659088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183918598 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83110903 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚587704310209536 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328925202 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚587704310209536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚643242841021440 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚564423853083840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8487629010468 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8518297818624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚564423836044480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992+mó '‡#…N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46088616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8519070476800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚587704310209536 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328925202 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚587704310209536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚643242841021440 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚564423853083840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8487629010468 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8518297818624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚564423836044480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚579185239732736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144983560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8518297818624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚572912252742928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129199519 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46089151 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144983560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8883948722176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8488399659088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183918598 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83110903 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚655870476318208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚655870256138752 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185008805 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129936153 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚591070063529984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚591070063529984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚582502518756864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚646934885877760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8567544773120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46352496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8537469104110 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8568317417984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83583657 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184985789 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚576198577540190 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚567661108436080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 trainer/global_step‚8349 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚582502536759808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884967781294072e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8935566500352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚655870476318208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚655870256138752 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185008805 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129936153 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚591070063529984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚591070063529984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚582502518756864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚646934885877760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8567544773120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46352496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8537469104110 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8568317417984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83583657 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184985789 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚576198577540190 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚567661108436080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 trainer/global_step‚8349 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚582502536759808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884967781294072e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8935566500352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83593303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129950536 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚582570236699136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8537699595576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚582570218696192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚591139558876160 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚582570236699136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145827262 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330857804 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚647010234220544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46357233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8568549518848 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145827262 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185030542 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚567727157832480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.fre½I×ed‚655946666290688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46357773 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8936622004736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚684 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚591139558876160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 trainer/global_step‚8349 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚576264857428056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145827260 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185030542 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚576265645116036 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 330857804 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8569322177024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚655946880165376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚591138768215040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846J…{F '™#„N VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 330857804 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8569322177024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚655946880165376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚591138768215040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚647010258160640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 330834760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8568549518848 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129951078 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83593303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129950536 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚582570236699136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8537699595576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚582570218696192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚591139558876160 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚582570236699136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145827262 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330857804 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚647010234220544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46357233 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8568549518848 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145827262 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185030542 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚567727157832480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚655946666290688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46357773 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8936622004736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚684 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚591139558876160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 trainer/global_step‚8349 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚576264857428056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145827260 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185030542 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚576265645116036 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992¬o¨ä2š#- _wandbruntime‚8878  _runtime‚8878LêÝæY›#Ó 0loss/FRt4/assignment_loss‚1.1432372946292162 2loss/SRqqt4/assignment_loss‚0.8826955630044852 /loss/FRt1/detection_loss‚2.3017322851753814 .loss/FBt2/detection_loss‚2.100889036288628 _step‚685 trainer/global_step‚8349 .loss/FRt4/detection_loss‚1.654634236707352 /loss/SRqqt3/detection_loss‚1.90199812669307 /loss/FRt1/assignment_loss‚2.691112236278813 0loss/FBt1/assignment_loss‚0.1354746756919973 1loss/FBt2/assignment_loss‚0.17030413373587058 1loss/FBt3/assignment_loss‚0.34085226841582994 &loss/total_loss‚42.028906074031696 0loss/FRt2/assignment_loss‚3.7118348864948048 0loss/FBt4/assignment_loss‚0.1114434591341916 .loss/FRt3/detection_loss‚2.844666017150437 2loss/SRqqt1/assignment_loss‚1.6529315909878775 .loss/FRt2/detection_loss‚2.361085156132129 1loss/SRqqt2/detection_loss‚2.7205114795102014 % _timestamp‚1.7884967790089326e+09 1loss/SRqqt2/assignment_loss‚1.823801966280573 2loss/SRqqt3/assignment_loss‚1.4365571002010256 1loss/SRqqt4/detection_loss‚0.8107888864264602 /loss/FBt1/detection_loss‚2.1395990206645084 /loss/FBt3/detection_loss‚2.9975520372390747 epoch‚6 1loss/SRqqt1/detection_loss‚2.6552776125344364 _runtime‚8878.649126317 /loss/FRt3/assignment_loss‚2.885106841723124 /loss/FBt4/detection_loss‚0.5548201629322078.ÛhI^œ#Ø trainer/global_step‚8349 .loss/FRt4/detection_loss‚1.654634236707352 /loss/SRqqt3/detection_loss‚1.90199812669307 /loss/FRt1/assignment_loss‚2.691112236278813 0loss/FBt1/assignment_loss‚0.1354746756919973 1loss/FBt2/assignment_loss‚0.17030413373587058 1loss/FBt3/assignment_loss‚0.34085226841582994 &loss/total_loss‚42.028906074031696 0loss/FRt2/assignment_loss‚3.7118348864948048 0loss/FBt4/assignment_loss‚0.1114434591341916 .loss/FRt3/detection_loss‚2.844666017150437 2loss/SRqqt1/assignment_loss‚1.6529315909878775 .loss/FRt2/detection_loss‚2.361085156132129 1loss/SRqqt2/detection_loss‚2.7205114795102014 % _timestamp‚1.7884967790089326e+09 1loss/SRqqt2/assignment_loss‚1.823801966280573 2loss/SRqqt3/assignment_loss‚1.4365571002010256 1loss/SRqqt4/detection_loss‚0.8107888864264602 /loss/FBt1/detection_loss‚2.1395990206645084 /loss/FBt3/detection_loss‚2.9975520372390747 epoch‚6 1loss/SRqqt1/detection_loss‚2.6552776125344364 _runtime‚8878.649126317 /loss/FRt3/assignment_loss‚2.885106841723124 /loss/FBt4/detection_loss‚0.5548201629322078 0loss/FRt4/assignment_loss‚1.1432372946292162 2loss/SRqqt4/assignment_loss‚0.8826955630044852 /loss/FRt1/detection_loss‚2.3017322851753814 .loss/FBt2/detection_loss‚2.100889036288628 _step‚685­z"#‚(:… Œ—éÔ¿Ú¸® gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.875' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡Þ «‰ž#‚(:þ Œ—éÔÒÓò) disk.nvme0c0n1.out‚210946.00341796875$ memory_percent‚93.17328929901123) proc.memory.availableMB‚ 1118.48828125 cpu‚25.276097158750297 network.sent‚ 2.0933513e+07) disk./.usagePercent‚78.76568650325002 disk.sda3.out‚ 590.94921875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613% disk./.usageGB‚1173.6656494140625! disk.nvme0n1.in‚ 29232.2890625$ disk.dm-0.out‚210946.00341796875 disk.drbd1260.out‚ 1.26171875 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd1260.in‚0 network.recv‚ 2.946826e+06' disk.nvme0n1.out‚210946.00341796875 disk.sda3.in‚0 disk.dm-1.out‚ 590.94921875 disk.sda.out‚ 590.94921875 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 29232.2890625 disk.dm-0.in‚ 29232.2890625 disk.sda2.out‚0š$+™‘Ÿ#‚(:† ›—éԜϱ gpu.0.gpu‚8.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.139' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.4278 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A·Õ… #‚(:ú ›—éÔßÓô) disk./.usagePercent‚78.76568368716715% disk./.usageGB‚1173.6656074523926 disk.sda.out‚ 591.32421875 disk.sda1.out‚0' disk.nvme0n1.out‚211299.53076171875& proc.memory.availableMB‚ 1119.46875* proc.memory.percent‚18.002820014953613 proc.cpu.threads‚17 network.sent‚ 2.1043133e+07 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 29232.2890625$ disk.dm-0.out‚211299.53076171875$ memory_percent‚93.16730499267578 network.recv‚ 2.952289e+06 disk.sda3.out‚ 591.32421875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.26171875 disk.sda.in‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚211299.53076171875! disk.nvme0n1.in‚ 29232.2890625 disk.dm-1.out‚ 591.32421875# disk.nvme0c0n1.in‚ 29232.2890625# proc.memory.rssMB‚ 2949.58203125 cpu‚25.27584679247124×$?¬¡#‚(:’ ª—éÔØ÷‚± gpu.0.gpu‚34.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.562599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hSæ¦{¢#‚(:ð ª—éÔ†——³ disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 1119.21875 network.recv‚ 2.955158e+06 disk.dm-0.in‚ 29232.2890625$ disk.dm-0.out‚211373.77001953125 disk.sda3.in‚0 disk.sda3.out‚ 591.953125 disk.dm-1.out‚ 591.953125 disk.drbd1260.out‚1.265625 disk.sda2.in‚0 network.sent‚ 2.1045489e+07# disk.nvme0c0n1.in‚ 29232.2890625! disk.nvme0n1.in‚ 29232.2890625 disk.sda.out‚ 591.953125 disk.sda2.out‚0$ memory_percent‚93.16883087158203# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613' disk.nvme0n1.out‚211373.77001953125 disk.drbd1260.in‚0 cpu‚25.27558928691167 proc.cpu.threads‚17) disk./.usagePercent‚78.75306507580444# disk./.usageGB‚1173.47758102417) disk.nvme0c0n1.out‚211373.77001953125ªbë’£#‚(:‡ ¹—éÔ±ª­® gpu.0.gpu‚38.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚165.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É\.¦~¤#‚(:ó ¹—éÔ‰Áà°! disk.nvme0n1.in‚ 29232.2890625$ disk.dm-0.out‚211886.12158203125 disk.dm-1.out‚ 592.5390625 disk.sda1.in‚0# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613 proc.cpu.threads‚17 network.sent‚ 2.1053507e+07' disk./.usagePercent‚78.803784264458' disk.nvme0n1.out‚211886.12158203125 disk.sda3.in‚0 disk.sda3.out‚ 592.5390625 disk.drbd1260.in‚0 disk.sda.out‚ 592.5390625$ memory_percent‚93.16730499267578 network.recv‚ 2.95973e+06% disk./.usageGB‚1174.2333335876465# disk.nvme0c0n1.in‚ 29232.2890625) disk.nvme0c0n1.out‚211886.12158203125 disk.dm-0.in‚ 29232.2890625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd1260.out‚1.265625 disk.sda1.out‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 1119.46875 cpu‚25.275366180189458ªÜ*Ë2¥#- _wandbruntime‚8928  _runtime‚8928¯±³¤Æ¦#À _step‚686 & lr-AdamW/pg1‚0.0002044329619645836 & lr-AdamW/pg2‚0.0002044329619645836 trainer/global_step‚8399 % _timestamp‚1.7884968287777538e+09 _runtime‚ 8928.41817472/ÁKM˧#Å & lr-AdamW/pg2‚0.0002044329619645836 trainer/global_step‚8399 % _timestamp‚1.7884968287777538e+09 _runtime‚ 8928.41817472 _step‚686 & lr-AdamW/pg1‚0.0002044329619645836®((fµ2¨#- _wandbruntime‚8928  _runtime‚8928±‘Ñl$©#õO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332728777 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚594506102844416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46619955 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚579551969913298 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚594505312196608 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84066057 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8587539689218 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8617796473344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8618569118208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332751795 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213888000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 186097707 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚594505312196608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46619406 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚594506102844416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚570964413184720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚585887515723264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚659690387475456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146654086 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚650702303016960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146654088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚650702326957056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186074691 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186097707 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146654086 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚659690601363456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8988274406400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8988084458496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚579551182225526 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚687 % _timestamp‚1.7884968287789104e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130685461z(ª#úO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8618569118208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332751795 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213888000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 186097707 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚594505312196608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46619406 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚594506102844416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚570964413184720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚585887515723264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚659690387475456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146654086 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚650702303016960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146654088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚650702326957056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186074691 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186097707 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146654086 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚659690601363456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8988274406400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8988084458496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚579551182225526 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚687 % _timestamp‚1.7884968287789104e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130685461 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332728777 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚594506102844416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46619955 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚579551969913298 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚594505312196608 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84066057 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8587539689218 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8617796473344¯ŒTŸí2«#- _wandbruntime‚8929  _runtime‚8929•ÚåÍ'¬#€N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46625292 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8587770180684 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚571030496659840 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186119444 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚594574807542784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186096402 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146670960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130700997 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46624740 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8989324570624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚585955215662592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚554 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8618801219072 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚594574016881664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 % _timestamp‚1.7884968296718483e+09 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚8929.312986101 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186119444 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚585955233665536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚585955233665536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332790406 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146670960 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8618801219072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚650777675299840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8989128344576 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130700443 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚579619037489144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332790406 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚659766999870464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚579618249801164 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚659766779704320 trainer/global_step‚8399 _step‚688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84075705 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146670962 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186096402 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚571030479620480 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115z1^ '­#…N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146670960 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8618801219072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚650777675299840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8989128344576 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130700443 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚579619037489144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332790406 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚659766999870464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚579618249801164 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚659766779704320 trainer/global_step‚8399 _step‚688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84075705 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146670962 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186096402 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚571030479620480 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46625292 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8587770180684 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚571030496659840 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186119444 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚594574807542784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186096402 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146670960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130700997 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46624740 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8989324570624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚585955215662592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚554 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8618801219072 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚594574016881664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 % _timestamp‚1.7884968296718483e+09 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚8929.312986101 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186119444 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚585955233665536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚585955233665536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332790406 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0°Í422®#- _wandbruntime‚8929  _runtime‚8929[ƒÐßW¯#Ñ 2loss/SRqqt3/assignment_loss‚1.3573990971213434 0loss/SRqqt2/detection_loss‚3.025254413485527 1loss/SRqqt4/detection_loss‚0.9261899823943774 trainer/global_step‚8399 % _timestamp‚1.7884968296746342e+09 _runtime‚8929.314812222 0loss/SRqqt1/detection_loss‚2.835042506456375 %loss/total_loss‚41.92098279249663 0loss/FRt1/assignment_loss‚3.0539509505033493 .loss/FRt1/detection_loss‚2.406882803680168 /loss/FRt3/assignment_loss‚2.877021833260854 1loss/SRqqt4/assignment_loss‚1.057462698417415 .loss/FRt3/detection_loss‚2.625607632348935 /loss/FRt2/assignment_loss‚3.495580615820708 0loss/FRt4/assignment_loss‚1.2419597836641165 2loss/SRqqt1/assignment_loss‚1.3436399172991513 1loss/FBt4/assignment_loss‚0.22164600529157039 /loss/FRt4/detection_loss‚1.6710161254383051 1loss/FBt2/assignment_loss‚0.16163389540135237 .loss/FBt1/detection_loss‚2.284587664736642 _step‚689 1loss/FBt3/assignment_loss‚0.06042119335869245 .loss/FRt2/detection_loss‚2.601971015974157 .loss/FBt2/detection_loss‚2.486130968729655 .loss/FBt3/detection_loss‚2.221207771450281 epoch‚6 1loss/SRqqt2/assignment_loss‚1.415392099879682 1loss/FBt1/assignment_loss‚0.22368116807738325 1loss/SRqqt3/detection_loss‚1.7456840387568242 /loss/FBt4/detection_loss‚0.5816186109497663«6™m°#Ö /loss/FRt3/assignment_loss‚2.877021833260854 1loss/SRqqt4/assignment_loss‚1.057462698417415 .¸®ÐÌïloss/FRt3/detection_loss‚2.625607632348935 /loss/FRt2/assignment_loss‚3.495580615820708 0loss/FRt4/assignment_loss‚1.2419597836641165 2loss/SRqqt1/assignment_loss‚1.3436399172991513 1loss/FBt4/assignment_loss‚0.22164600529157039 /loss/FRt4/detection_loss‚1.6710161254383051 1loss/FBt2/assignment_loss‚0.16163389540135237 .loss/FBt1/detection_loss‚2.284587664736642 _step‚689 1loss/FBt3/assignment_loss‚0.06042119335869245 .loss/FRt2/detection_loss‚2.601971015974157 .loss/FBt2/detection_loss‚2.486130968729655 .loss/FBt3/detection_loss‚2.221207771450281 epoch‚6 1loss/SRqqt2/assignment_loss‚1.415392099879682 1loss/FBt1/assignment_loss‚0.22368116807738325 1loss/SRqqt3/detection_loss‚1.7456840387568242 /loss/FBt4/detection_loss‚0.5816186109497663 2loss/SRqqt3/assignment_loss‚1.3573990971213434 0loss/SRqqt2/detection_loss‚3.025254413485527 1loss/SRqqt4/detection_loss‚0.9261899823943774 trainer/global_step‚8399 % _timestamp‚1.7884968296746342e+09 _runtime‚8929.314812222 0loss/SRqqt1/detection_loss‚2.835042506456375 %loss/total_loss‚41.92098279249663 0loss/FRt1/assignment_loss‚3.0539509505033493 .loss/FRt1/detection_loss‚2.406882803680168±at¤±#‚(:… È—éÔ·ž³° gpu.0.gpu‚35.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 $ _timestamp‚1.788496882860172e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46887660 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚597941351511040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚663510561672704 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187186609 trainer/global_step‚8449 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚582905362286406 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8637610274326 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131436683 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131436115 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚8982.500675059 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8668820818432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46888226 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9041023075840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334684397 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147497786 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚574267752012080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147497788 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187186609 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚663510767172096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8668048173568 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚597940560863232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664Oñi˜(¼#úO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205499392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147497788 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚589272512689664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 _step‚691 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚589272512689664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8668048173568 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚589272530692608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚589272530692608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚654469720156160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚582905362286406 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8637610274326 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131436683 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131436115 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚8982.500675059 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8668820818432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46888226 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9041023075840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334684397 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147497786 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚574267752012080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147497788 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187186609 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚663510767172096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8668048173568 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚597940560863232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚654469744096256 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9040841516544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84548455 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147497786 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 $ _timestamp‚1.788496882860172e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46887660 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚597941351511040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚663510561672704 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187186609 trainer/global_step‚8449 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8669052919296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187208346 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚582972429862252 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8637840765792 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚589340212628992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147514662 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚582971642174272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8669825577472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334723008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228554752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131450984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334699964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚692 OADeviceStatsMãA¯ Monitor.on_train_batch_end/allocation.large_pool.freed‚ 147514660 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334699964 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚598009265548288 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84558103 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8669052919296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187208346 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚582972429862252 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8637840765792 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚589340212628992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147514662 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚582971642174272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8669825577472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334723008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228554752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131450984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334699964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚692 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147514660 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334699964 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚598009265548288 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042´Ë"šo2À#- _wandbruntime‚8983  _runtime‚8983Zçî\Á#Ö 0loss/FRt3/assignment_loss‚2.9067299113129126 2loss/SRqqt2/assignment_loss‚1.9251211149537044 0loss/FBt1/assignment_loss‚0.2838609222438241 .loss/FBt2/detection_loss‚2.421332130065331 _step‚693 /loss/FRt2/assignment_loss‚2.756834926812545 1loss/FBt3/assignment_loss‚0.14422741976173872 1loss/SRqqt3/detection_loss‚1.9766852499350258 /loss/FBt1/detection_loss‚2.7534355436052596 epoch‚6 _runtime‚8983.393642787 2loss/SRqqt3/assignment_loss‚1.5244821066279775 1loss/FBt2/assignment_loss‚0.22703132075429486 %loss/total_loss‚41.96072337559188 2loss/SRqqt1/assignment_loss‚1.8507753644759457 /loss/FRt1/detection_loss‚2.3411718338001064 .loss/FRt3/detection_loss‚2.580902034586126 .loss/FRt4/detection_loss‚1.512615516036749 1loss/SRqqt4/detection_loss‚0.9333123481289173 % _timestamp‚1.7884968837535071e+09 1loss/SRqqt2/detection_loss‚2.8475266720937644 0loss/FRt4/assignment_loss‚1.0050590974943978 2loss/SRqqt4/assignment_loss‚0.9573741049001304 1loss/SRqqt1/detection_loss‚2.1135355730851493 /loss/FBt3/detection_loss‚2.1826500682269825 trainer/global_step‚8449 /loss/FRt1/assignment_loss‚3.037869880839092 0loss/FBt4/assignment_loss‚0.1896239647781361 /loss/FRt2/detection_loss‚3.0002818263510203 /loss/FBt4/detection_loss‚0.4882844447227497ÕÜ]ÐaÂ#Û 2loss/SRqqt1/assignment_loss‚1.8507753644759457 /loss/FRt1/detection_loss‚2.3411718338001064 .loss/FRt3/detection_loss‚2.580902034586126 .loss/FRt4/detection_loss‚1.512615516036749 1loss/SRqqt4/detection_loss‚0.9333123481289173 % _timestamp‚1.7884968837535071e+09 1loss/SRqqt2/detection_loss‚2.8475266720937644 0loss/FRt4/assignment_loss‚1.0050590974943978 2loss/SRqqt4/assignment_loss‚0.9573741049001304 1loss/SRqqt1/detection_loss‚2.1135355730851493 /loss/FBt3/detection_loss‚2.1826500682269825 trainer/global_step‚8449 /loss/FRt1/assignment_loss‚3.037869880839092 0loss/FBt4/assignment_loss‚0.1896239647781361 /loss/FRt2/detection_loss‚3.0002818263510203 /loss/FBt4/detection_loss‚0.4882844447227497 0loss/FRt3/assignment_loss‚2.9067299113129126 2loss/SRqqt2/assignment_loss‚1.9251211149537044 0loss/FBt1/assignment_loss‚0.2838609222438241 .loss/FBt2/detection_loss‚2.421332130065331 _step‚693 /loss/FRt2/assignment_loss‚2.756834926812545 1loss/FBt3/assignment_loss‚0.14422741976173872 1loss/SRqqt3/detection_loss‚1.9766852499350258 /loss/FBt1/detection_loss‚2.7534355436052596 epoch‚6 _runtime‚8983.393642787 2loss/SRqqt3/assignment_loss‚1.5244821066279775 1loss/FBt2/assignment_loss‚0.22703132075429486 %loss/total_loss‚41.96072337559188µ.ûÃ#‚(:’ õ—éÔ’­ò² gpu.0.gpu‚38.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.427' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.885399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+¸ÎÉ{Ä#‚(:ð õ—éÔî׌´ disk.sda3.out‚ 594.578125 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.2734375 disk.sda2.in‚0 cpu‚25.274351525750866 network.recv‚ 2.977992e+06# disk.nvme0c0n1.in‚ 29355.2890625 disk.dm-0.in‚ 29355.2890625 disk.dm-1.out‚ 594.578125 disk.sda.in‚0 disk.sda1.out‚0# memory_percent‚93.1711196899414 network.sent‚ 2.1176652e+07) disk./.usagePercent‚78.80562419060658 disk.drbd1260.in‚0 disk.sda.out‚ 594.578125 disk.sda1.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2949.58203125 proc.cpu.threads‚17& disk.nvme0n1.out‚213048.9833984375# disk.dm-0.out‚213048.9833984375 disk.sda3.in‚0& proc.memory.availableMB‚ 1118.84375* proc.memory.percent‚18.002820014953613% disk./.usageGB‚1174.2607498168945( disk.nvme0c0n1.out‚213048.9833984375! disk.nvme0n1.in‚ 29355.2890625lͬ’Å#‚(:‡ „˜éÔ«öˆ® gpu.0.gpu‚56.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚178.953' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7906 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(¬jvÆ#‚(:ë „˜éÔÅ–±¯ disk.drbd1260.in‚0 disk.sda.in‚0" disk.nvme0n1.in‚29466.87109375" disk.nvme0n1.out‚ 213155.515625 disk.dm-0.out‚ 213155.515625# proc.memory.rssMB‚ 2949.58203125 network.recv‚ 2.984035e+06% disk./.usageGB‚1174.2684364318848 disk.sda3.in‚0 disk.sda3.out‚ 595.140625 disk.dm-1.in‚0 disk.dm-1.out‚ 595.140625* proc.memory.percent‚18.002820014953613 cpu‚25.27555019420037 disk.drbd1260.out‚ 1.27734375 disk.sda1.in‚0 disk.dm-0.in‚29466.87109375$ memory_percent‚93.17419528961182) proc.memory.availableMB‚ 1118.33984375 proc.cpu.threads‚17 network.sent‚ 2.1285119e+07) disk./.usagePercent‚78.80614004578807 disk.sda.out‚ 595.140625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚29466.87109375$ disk.nvme0c0n1.out‚ 213155.515625€ '©Ç#‚(:„ “˜éÔ‡úÑ® gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œë'qÈ#‚(:æ “˜éÔ±ó›¯ disk.sda2.out‚0" disk.nvme0n1.in‚29466.87109375 disk.dm-0.out‚ 213387.859375 disk.sda3.out‚ 595.546875 disk.sda1.out‚0$ memory_percent‚93.16825866699219 cpu‚25.28185831307129( disk./.usagePercent‚78.8061387657504 disk.sda2.in‚0 disk.dm-0.in‚29466.87109375 disk.dm-1.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613 proc.cpu.threads‚17 network.sent‚ 2.1290233e+07$ disk.nvme0c0n1.out‚ 213387.859375 disk.drbd1260.out‚ 1.27734375 disk.sda.in‚0 disk.sda.out‚ 595.546875% disk./.usageGB‚1174.2684173583984$ disk.nvme0c0n1.in‚29466.87109375" disk.nvme0n1.out‚ 213387.859375 disk.sda3.in‚0 disk.dm-1.out‚ 595.546875 disk.drbd1260.in‚0% proc.memory.availableMB‚ 1119.3125 network.recv‚ 2.987864e+06⌆œÉ#‚(:‘ ¢˜éÔøž¥­ gpu.0.gpu‚51.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚130.15' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.030000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05;ý €Ê#‚(:õ ¢˜éÔÛº¯ proc.cpu.threads‚17 network.recv‚ 2.99116e+06 disk.sda1.out‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚29466.87109375 disk.dm-0.in‚29466.87109375$ memory_percent‚93.16980838775635# proc.memory.rssMB‚ 2949.58203125 cpu‚25.29025658924117( disk./.usagePercent‚78.8061387657504 disk.drbd1260.in‚0 disk.drbd1260.out‚1.28125 disk.sda2.out‚0# disk.dm-0.out‚213718.3525390625 disk.sda3.out‚ 596.0390625 disk.dm-1.out‚ 596.0390625 network.sent‚ 2.1295242e+07% disk./.usageGB‚1174.2684173583984 disk.sda.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚213718.3525390625" disk.nvme0n1.in‚29466.87109375& disk.nvme0n1.out‚213718.3525390625 disk.sda.out‚ 596.0390625 disk.sda3.in‚0 disk.dm-1.in‚0) proc.memory.availableMB‚ 1119.05859375* proc.memory.percent‚18.002820014953613còLË#‚(:’ ±˜éÔŸ¼ï¯ gpu.0.gpu‚53.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚132.338' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.467599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,i2 ~Ì#‚(:ó ±˜éÔèÊû²$ memory_percent‚93.16983222961426# proc.memory.rssMB‚ 2949.58203125* proc.memory.percent‚18.002820014953613 cpu‚25.29575918014814 network.sent‚ 2.1301594e+07) disk./.usagePercent‚78.81473268266731$ disk.nvme0c0n1.in‚29466.87109375 disk.sda3.in‚0 disk.drbd1260.out‚1.28125 disk.sda.out‚ 596.65625 disk.sda1.in‚0% disk./.usageGB‚1174.3964729309082" disk.nvme0n1.in‚29466.87109375' disk.nvme0n1.out‚214073.13525390625 disk.sda3.out‚ 596.65625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 596.65625 network.recv‚ 2.996141e+06) disk.nvme0c0n1.out‚214073.13525390625 disk.dm-0.in‚29466.87109375$ disk.dm-0.out‚214073.13525390625 disk.sda.in‚0( proc.memory.availableMB‚ 1119.0546875 proc.cpu.threads‚17 disk.drbd1260.in‚0Iêœã2Í#- _wandbruntime‚9055  _runtime‚9055Ù׺UX&Î#ÝT NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚47210432 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 336247939 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18215 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 148369652 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111864 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚601153339456512 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 124598784 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18217 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚592448243802624 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 336247939 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 132284794 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚577265178277424 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚657826550013952 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634825 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚516 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28115 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 148369650 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚666905113791488 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1005 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18215 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12155092992 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 148538880 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8705095653888 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 187878289 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27846 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 9078712316416 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚577265161238064 _step‚694 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚601153339456512 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 132284981 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 trainer/global_step‚8462 _runtime‚9055.710365406 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 336266156 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚85074549 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚585939516494938 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 10929229312 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 148369652 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18217 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚187 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚185 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚666905262330368 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 % _timestamp‚1.7884969560693738e+09 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0;‚Th*Ï#âT _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚666905113791488 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1005 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18215 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12155092992 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 148538880 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8705095653888 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 187878289 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27846 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 9078712316416 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚577265161238064 _step‚694 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚601153339456512 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 132284981 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 trainer/global_step‚8462 _runtime‚9055.710365406 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 336266156 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚85074549 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚585939516494938 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 10929229312 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 148369652 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18217 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚187 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚185 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚666905262330368 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 % _timestamp‚1.7884969560693738e+09 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚47210432 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 336247939 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18215 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 148369652 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111864 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚601153339456512 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 124598784 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18217 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚592448243802624 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 336247939 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 132284794 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚577265178277424 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚657826550013952 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634825 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚516 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28115 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 148369650¶ÆÞÏ32Ð#- _wandbruntime‚9056  _runtime‚9056‡Ù/m)Ñ#çR WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12155092992 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8705560402432 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 132297908 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1046 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1005 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚577307457078160 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1019674624 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚666952556685312 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 148382622 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18215 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27846 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚601199272204288 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580044288 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 598047232 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚85082527 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18215 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 148382624 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 8674801378478 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12155092992 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚592493729804800 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 336287422 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚187 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚85082529 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚39 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 10558147584 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 _step‚695 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚601199272204288 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚47215381 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚585981680384134 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8706140446720 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18217 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 336269205 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 580044288 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 375175680 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8674223305974 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 187904798 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9079052466176 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595111864 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚516 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18217 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 148382622 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 187886583 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚577307474117520 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28115 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚592493711801856 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 trainer/global_step‚8462 _runtime‚9056.954915353 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚666952705224192 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚585982275495998 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚516 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚601199870251520 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚47215566 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 11518882825 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1017634825 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 336269205 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892004352 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 9079177064960 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1005 % _timestamp‚1.7884969573141437e+09 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 598047232 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8705560402432 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 187904798 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 10929229312 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27846 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 148538880 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚657873528159232 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1019674624 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚666952556685312 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 148382622 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18215 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚27846 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚601199272204288 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580044288 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 598047232 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚85082527 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚18215 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 148382624 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 8674801378478 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 12155092992 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚592493729804800 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 336287422 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚187 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚85082529 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚39 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 10558147584 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 _step‚695 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚601199272204288 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚47215381 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚585981680384134 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8706140446720 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚18217 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 336269205 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 580044288 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 375175680 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8674223305974 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 187904798 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9079052466176 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595111864 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚516 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚18217 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 148382622 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 187886583 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚577307474117520 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peakVµÙY§‚28115 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚592493711801856 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 trainer/global_step‚8462 _runtime‚9056.954915353 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚666952705224192·¢;’µ‘Ó#‚(:† À˜éÔïä® gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚175.526' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.1052 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\;X3Ô#‚(:ö À˜éÔÖ Ž³ disk.drbd1260.out‚ 1.28515625% disk./.usageGB‚1174.5220108032227 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚93.17307472229004( proc.memory.availableMB‚ 1118.5234375# proc.memory.rssMB‚ 2949.58203125$ disk.nvme0c0n1.in‚29585.41796875% disk.nvme0n1.out‚214426.943359375 disk.drbd1260.in‚0 disk.sda.in‚0* proc.memory.percent‚18.002820014953613 network.sent‚ 2.1307512e+07 disk.dm-0.in‚29585.41796875 disk.sda3.out‚ 597.0390625 disk.sda.out‚ 597.0390625 disk.sda2.out‚0 cpu‚25.30098945328063 proc.cpu.threads‚17 disk.dm-1.out‚ 597.0390625 network.recv‚ 3.000728e+06) disk./.usagePercent‚78.82315763461163' disk.nvme0c0n1.out‚214426.943359375" disk.nvme0n1.in‚29585.41796875" disk.dm-0.out‚214426.943359375 disk.sda3.in‚05va2Õ#- _wandbruntime‚9073  _runtime‚9073]{^ƒòxÖ#‘¨ 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887555241584778 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0.5 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.3125 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9700972437858582 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4856554865837097 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ,particle/sensitivity‚0.31875357031822205 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0306021049618721 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.21604938805103302 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20425190031528473 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.02984379231929779 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.689393937587738 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10842897742986679 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.473920077085495 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6650641560554504 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39444446563720703 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2686392366886139 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5313858985900879 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 +jet/accuracy_4_of_6‚0.03846153989434242 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05496896058320999 -jet/accuracy_5_of_5‚0.0026881720405071974 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4499548673629761 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_2‚0.7367604374885559 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.45095399022102356 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9977035522460938 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12015120685100555 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 +jet/accuracy_3_of_5‚0.11504096537828445 particle/accuracy_3_of_4‚1 particle/accuracy_1_of_6‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16198699176311493 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983808994293213 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2606363892555237 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 'particle/accuracy‚0.849289059638977 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18636535108089447 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9885072708129883 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48520591855049133 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6014994978904724 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19412830471992493 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2712487280368805 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3308466970920563 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12627333402633667 particle/accuracy_2_of_5‚1 *jet/accuracy_2_of_6‚0.4615384638309479 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.40751639008522034 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2993881404399872 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4074612259864807 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4837963283061981 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2747959792613983 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23482756316661835 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6974206566810608 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.25555557012557983 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9855167269706726 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47732365131378174 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 )loss/val_total_loss‚39.96790921400133 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22621513903141022 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.037144120782613754 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4285714328289032 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5007762312889099 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22575776278972626 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +particle/specificity‚0.9474677443504333 jet/accuracy_6_of_6‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.06666666269302368 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9005847573280334 8"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.555555522441864 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09394766390323639 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.003448275849223137 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23085586726665497 +jet/accuracy_3_of_4‚0.07978889346122742 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4374999701976776 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.45362788438796997 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0020502645056694746 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.45896992087364197 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 particle/accuracy_3_of_3‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5598405003547668 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4910670220851898 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9843447804450989 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.34634923934936523 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9615384340286255 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4226083755493164 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_3_of_6‚0.1538461595773697 jet/accuracy_6_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20305348932743073 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5669370293617249 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3270200192928314 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21212120354175568 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19998492300510406 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49143165349960327 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4177388846874237 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5936747789382935 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 trainer/global_step‚8462 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8206178545951843 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.20567373931407928 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9219858646392822 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981447458267212 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.46639785170555115 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9897336959838867 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 6"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4511698782444 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3916666805744171 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9338625073432922 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9882873296737671 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.42715781927108765 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4019608199596405 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9300000071525574 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24913792312145233 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,jet/accuracy_4_of_4‚0.009515573270618916 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.035402558743953705 +jet/accuracy_1_of_1‚0.47644752264022827 particle/accuracy_3_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 _step‚696 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.6229007244110107 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20434148609638214 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.09441708028316498 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9957482814788818 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981447458267212 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48405972123146057 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23482756316661835 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013759169727563858 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1510416567325592 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48714184761047363 5 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20689657330513 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3958333730697632 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.3125 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4852738678455353 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14406712353229523 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0013227513991296291 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_4_of_5‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.046294160187244415 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1854175478219986 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.380447119474411 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5574713349342346 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 'particle/f_score‚0.3975756764411926 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7456140518188477 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.986736536026001 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18466980755329132 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.36770835518836975 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_3‚0.8538058996200562 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9969592094421387 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666865348816 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.6061886548995972 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.41329967975616455 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4305555522441864 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2301587164402008 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4645549952983856 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.41941383481025696 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18677283823490143 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6254053115844727 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.979082465171814 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9868984818458557 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02269291877746582 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.6000000238418579 particle/accuracy_5_of_7‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26975157856941223 particle/accuracy_2_of_6‚1 particle/accuracy_4_of_7‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17938274145126343 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7045454382896423 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5037162899971008 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1766534447669983 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_5_of_6‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.35654571652412415 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9795114994049072 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1508771926164627 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4965278208255768 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.26680976152420044 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975086450576782 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.073248490691185 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4303639829158783 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.41069090366363525 7validation_average_jet_accuracy‚0.43343788385391235 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9340277910232544 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020546739920973778 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_3‚0.02998991869390011 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.444444477558136 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.6100000143051147 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6398808360099792 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969176650047302 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.49515506625175476 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 _runtime‚9073.126186842 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39351844787597656 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9843345284461975 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.6068350076675415 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4611913859844208 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3712121248245239 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4695255756378174 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.64762943983078 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2321428507566452 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24482160806655884 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.46225830912590027 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3037037253379822 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.40019890666007996 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6224782466888428 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6093416213989258 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9813492298126221 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_5‚0.9367896318435669 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37459513545036316 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5034524202346802 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 particle/accuracy_4_of_4‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11487266421318054 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.2777777910232544 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4888298511505127 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666865348816 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9715437889099121 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 epoch‚6 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4512155055999756 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20929843187332153 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5668860077857971 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24368688464164734 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9914907217025757 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9992063641548157 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08660713583230972 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884969734822536e+09 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6453709602355957 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6378471851348877 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26932471990585327 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4244832396507263 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03211973235011101 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26855728030204773 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.270294189453125 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07566408067941666 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_2‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.3202160596847534 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_5_of_6‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3319386839866638 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008118247613310814 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3810846507549286 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9626436829566956 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046*Õc¬&126365662 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2413114756345749 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5559298992156982 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4512155055999756 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5658046007156372 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9874902367591858 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.12719297409057617 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.708677351474762 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9947922229766846 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47582563757896423 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.32406264543533325 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17835669219493866 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5496850609779358 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19818681478500366 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9897661209106445 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9874843955039978 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9212172031402588 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 particle/accuracy_2_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9888378977775574 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.21638213098049164 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,jet/accuracy_4_of_5‚0.018612390384078026 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2551625669002533 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.43455269932746887 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.3125 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21178613603115082 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.48736774921417236 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 particle/accuracy_1_of_7‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6009808778762817 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1640804558992386 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16994206607341766 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3647308945655823 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36353984475135803 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23099587857723236 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5194725394248962 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4575493037700653 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25322863459587097 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2407449632883072 particle/accuracy_3_of_6‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.6042574048042297 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.029026539996266365 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.45206424593925476 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5370370149612427 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.43333330750465393 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.570002555847168 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.43243247270584106 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16844911873340607 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.424566388130188ó<õÌd×#–¨ 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5194725394248962 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4575493037700653 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25322863459587097 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2407449632883072 particle/accuracy_3_of_6‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.6042574048042297 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.029026539996266365 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.45206424593925476 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5370370149612427 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.43333330750465393 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.570002555847168 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.43243247270584106 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16844911873340607 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.424566388130188 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887555241584778 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0.5 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.3125 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9700972437858582 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4856554865837097 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ,particle/sensitivity‚0.31875357031822205 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0306021049618721 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.21604938805103302 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20425190031528473 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.02984379231929779 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.689393937587738 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10842897742986679 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.473920077085495 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6650641560554504 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39444446563720703 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2686392366886139 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5313858985900879 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 +jet/accuracy_4_of_6‚0.03846153989434242 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05496896058320999 -jet/accuracy_5_of_5‚0.0026881720405071974 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4499548673629761 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_2‚0.7367604374885559 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.45095399022102356 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9977035522460938 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12015120685100555 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 +jet/accuracy_3_of_5‚0.11504096537828445 particle/accuracy_3_of_4‚1 particle/accuracy_1_of_6‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16198699176311493 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983808994293213 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2606363892555237 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 'particle/accuracy‚0.849289059638977 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18636535108089447 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9885072708129883 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48520591855049133 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6014994978904724 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19412830471992493 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2712487280368805 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3308466970920563 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12627333402633667 particle/accuracy_2_of_5‚1 *jet/accuracy_2_of_6‚0.4615384638309479 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.40751639008522034 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2993881404399872 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4074612259864807 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4837963283061981 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2747959792613983 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23482756316661835 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6974206566810608 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.25555557012557983 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9855167269706726 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47732365131378174 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 )loss/val_total_loss‚39.96790921400133 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22621513903141022 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.037144120782613754 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4285714328289032 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5007762312889099 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22575776278972626 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +particle/specificity‚0.9474677443504333 jet/accuracy_6_of_6‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.06666666269302368 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9005847573280334 8"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.555555522441864 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09394766390323639 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.003448275849223137 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23085586726665497 +jet/accuracy_3_of_4‚0.07978889346122742 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4374999701976776 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.45362788438796997 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0020502645056694746 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.45896992087364197 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 particle/accuracy_3_of_3‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5598405003547668 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4910670220851898 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9843447804450989 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.34634923934936523 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9615384340286255 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4226083755493164 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_3_of_6‚0.1538461595773697 jet/accuracy_6_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20305348932743073 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5669370293617249 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3270200192928314 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21212120354175568 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19998492300510406 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49143165349960327 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4177388846874237 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5936747789382935 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 trainer/global_step‚8462 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8206178545951843 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.20567373931407928 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9219858646392822 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981447458267212 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.46639785170555115 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9897336959838867 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 6"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4511698782444 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3916666805744171 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9338625073432922 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9882873296737671 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.42715781927108765 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4019608199596405 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9300000071525574 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24913792312145233 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,jet/accuracy_4_of_4‚0.009515573270618916 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.035402558743953705 +jet/accuracy_1_of_1‚0.47644752264022827 particle/accuracy_3_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 _step‚696 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.6229007244110107 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20434148609638214 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.09441708028316498 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9957482814788818 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981447458267212 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48405972123146057 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23482756316661835 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013759169727563858 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1510416567325592 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48714184761047363 5 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20689657330513 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3958333730697632 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.3125 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4852738678455353 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14406712353229523 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0013227513991296291 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_4_of_5‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.046294160187244415 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1854175478219986 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.380447119474411 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5574713349342346 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 'particle/f_score‚0.3975756764411926 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7456140518188477 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.986736536026001 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18466980755329132 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.36770835518836975 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_3‚0.8538058996200562 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9969592094421387 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666865348816 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.6061886548995972 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.41329967975616455 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4305555522441864 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2301587164402008 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4645549952983856 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.41941383481025696 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18677283823490143 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6254053115844727 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.979082465171814 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9868984818458557 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02269291877746582 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.6000000238418579 particle/accuracy_5_of_7‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26975157856941223 particle/accuracy_2_of_6‚1 particle/accuracy_4_of_7‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17938274145126343 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7045454382896423 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5037162899971008 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1766534447669983 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_5_of_6‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.35654571652412415 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9795114994049072 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1508771926164627 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4965278208255768 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.26680976152420044 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975086450576782 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.073248490691185 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4303639829158783 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.41069090366363525 7validation_average_jet_accuracy‚0.43343788385391235 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9340277910232544 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020546739920973778 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_3‚0.02998991869390011 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.444444477558136 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.6100000143051147 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6398808360099792 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969176650047302 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.49515506625175476 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 _runtime‚9073.126186842 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39351844787597656 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_pu’*ïdQ/rity‚0.5 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9843345284461975 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.6068350076675415 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4611913859844208 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3712121248245239 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4695255756378174 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.64762943983078 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2321428507566452 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24482160806655884 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.46225830912590027 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3037037253379822 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.40019890666007996 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6224782466888428 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6093416213989258 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9813492298126221 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_5‚0.9367896318435669 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37459513545036316 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5034524202346802 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 particle/accuracy_4_of_4‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11487266421318054 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.2777777910232544 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4888298511505127 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666865348816 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9715437889099121 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 epoch‚6 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4512155055999756 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20929843187332153 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5668860077857971 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24368688464164734 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9914907217025757 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9992063641548157 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08660713583230972 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884969734822536e+09 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6453709602355957 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6378471851348877 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26932471990585327 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4244832396507263 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03211973235011101 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26855728030204773 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.270294189453125 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07566408067941666 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_2‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.3202160596847534 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_5_of_6‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3319386839866638 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008118247613310814 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3810846507549286 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9626436829566956 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2413114756345749 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5559298992156982 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4512155055999756 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5658046007156372 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9874902367591858 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.12719297409057617 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.708677351474762 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9947922229766846 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47582563757896423 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.32406264543533325 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17835669219493866 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5496850609779358 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19818681478500366 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9897661209106445 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9874843955039978 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9212172031402588 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 particle/accuracy_2_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9888378977775574 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.21638213098049164 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,jet/accuracy_4_of_5‚0.018612390384078026 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2551625669002533 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.43455269932746887 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.3125 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21178613603115082 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.48736774921417236 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 particle/accuracy_1_of_7‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6009808778762817 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1640804558992386 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16994206607341766 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3647308945655823 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36353984475135803 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23099587857723236 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0¸Sö™Ø#‚(:… ϘéÔçÊ»° gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7348 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç5]x„Ù#‚(:ù ϘéÔ¹½Àµ disk.sda3.out‚ 597.51953125 disk.dm-1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 601.45703125" proc.memory.rssMB‚ 3466.8046875* proc.memory.percent‚21.159696578979492 network.sent‚ 2.1414839e+07 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 597.51953125 disk.sda2.out‚0' disk.nvme0c0n1.out‚214683.275390625" disk.nvme0n1.in‚29585.41796875 disk.dm-0.in‚29585.41796875) disk./.usagePercent‚78.81463488778924 disk.sda3.in‚0 disk.dm-1.out‚ 597.51953125 disk.drbd1260.out‚ 1.28515625 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚29585.41796875" disk.dm-0.out‚214683.275390625% disk.nvme0n1.out‚214683.275390625$ memory_percent‚96.32899761199951 cpu‚25.305561378894556 proc.cpu.threads‚17 network.recv‚ 3.006219e+06% disk./.usageGB‚1174.3950157165527Š?äæ’Ú#‚(:‡ Þ˜éÔ¬–º² gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚170.514' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡öæê|Û#‚(:ñ Þ˜éÔ®Œÿ³& disk.nvme0n1.out‚216214.5556640625 disk.drbd1260.out‚ 1.2890625 disk.sda.in‚0 disk.sda.out‚ 597.7890625 disk.sda1.out‚0 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚89.76364135742188$ proc.memory.availableMB‚1677.125 cpu‚25.30528864393562 proc.cpu.threads‚17 network.sent‚ 2.1538445e+07 network.recv‚ 3.012044e+06 disk.sda3.in‚0 disk.sda3.out‚ 597.7890625 disk.drbd1260.in‚0! proc.memory.rssMB‚ 2949.640625( disk./.usagePercent‚78.8145665337776" disk.nvme0n1.in‚29585.41796875 disk.dm-0.in‚29585.41796875# disk.dm-0.out‚216214.5556640625 disk.dm-1.in‚0 disk.dm-1.out‚ 597.8203125 disk.sda2.in‚0* proc.memory.percent‚18.003177642822266% disk./.usageGB‚1174.3939971923828$ disk.nvme0c0n1.in‚29585.41796875( disk.nvme0c0n1.out‚216214.5556640625?;ŒÝ’Ü#‚(:‡ í˜éÔ“ÒÛ° gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚116.868' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@Ï€Ý#‚(:õ í˜éÔ„¡à²( disk.nvme0c0n1.out‚216565.8720703125 disk.dm-0.in‚29585.41796875 disk.sda3.in‚0( disk./.usagePercent‚78.8129949035249 disk.dm-1.out‚ 598.5859375 disk.sda.out‚ 598.57421875$ disk.nvme0c0n1.in‚29585.41796875# disk.dm-0.out‚216565.8720703125 disk.sda3.out‚ 598.5859375# memory_percent‚89.7517204284668 cpu‚25.304987781156402 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.in‚29585.41796875& disk.nvme0n1.out‚216565.8720703125' proc.memory.availableMB‚ 1679.078125! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 network.sent‚ 2.1544714e+07% disk./.usageGB‚1174.3705787658691 disk.dm-1.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 network.recv‚ 3.017108e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.2890625 disk.sda1.in‚0 disk.sda2.in‚0ŽÅQ2Þ#- _wandbruntime‚9115  _runtime‚9115ä.k,Çß#Á & lr-AdamW/pg1‚0.0001945056119739326 & lr-AdamW/pg2‚0.0001945056119739326 trainer/global_step‚8499 % _timestamp‚1.7884970160113287e+09 _runtime‚9115.651669479 _step‚697>šÎÌà#Æ & lr-AdamW/pg1‚0.0001945056119739326 & lr-AdamW/pg2‚0.0001945056119739326 trainer/global_step‚8499 % _timestamp‚1.7884970160113287e+09 _runtime‚9115.651669479 _step‚697¹+@ù˜2á#- _wandbruntime‚9115  _runtime‚9115^þñrü'â#öO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9122964492800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚9115.652866182 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604269589630464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚661196784388096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚604270380278272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8716877107544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚580235711726128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚661196760448000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚604270380278272 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188799970 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8747596022784 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149158722 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 % _timestamp‚1.7884970160123262e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337958694 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚670319530798592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133002032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚588951801145900 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8747596022784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133001477 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 trainer/global_step‚8499 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149158722 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚580235694686768 _step‚698@ɱ:-ã#ûO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚604270380278272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8716877107544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚580235711726128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚661196760448000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚604270380278272 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188799970 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8747596022784 M?DeviceStatsMonitor.oËÈ5Ôn_train_batch_start/active.large_pool.freed‚ 149158722 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 % _timestamp‚1.7884970160123262e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337958694 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚670319530798592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133002032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚588951801145900 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8747596022784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133001477 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 trainer/global_step‚8499 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149158722 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚580235694686768 _step‚698 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9122964492800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚9115.652866182 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604269589630464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚661196784388096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ºÇÅòq2ä#- _wandbruntime‚9116  _runtime‚9116Ìô;'å#€N E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337974261 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149175598 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188821707 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8748600768512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚589019656409518 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133016355 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚670396139972096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9123804723712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8717878247630 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚604339084976640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188798665 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188821707 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149175596 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8749373426688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47473651 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚661272132730880 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85543243 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 % _timestamp‚1.7884970170421877e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚580301761122528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚595589693547008 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133016896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149175596 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47473112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚595589711549952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85543245 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚699 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337997305 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚595589711549952 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337997305 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 trainer/global_step‚8499 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188798665 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149175598 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚604338294315520 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚604339084976640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8717107599010 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8748600768512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9124007241216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚589018868721538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚9116.683925069 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚670395913514496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚580301778161888 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992öã±8 'æ#…N P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚604338294315520 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚604339084976640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8717107599010 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8748600768512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9124007241216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚589018868721538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚9116.683925069 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚670395913514496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚580301778161888 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337974261 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149175598 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188821707 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8748600768512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚589019656409518 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133016355 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚670396139972096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9123804723712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8717878247630 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚604339084976640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188798665 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188821707 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149175596 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8749373426688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47473651 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚661272132730880 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85543243 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 % _timestamp‚1.7884970170421877e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚580301761122528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚595589693547008 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133016896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149175596 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47473112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚595589711549952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85543245 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚699 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337997305 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚595589711549952 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337997305 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 trainer/global_step‚8499 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188798665 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149175598 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0»³²k;2ç#- _wandbruntime‚9116  _runtime‚9116²}@\è#Ö 2loss/SRqqt2/assignment_loss‚1.4695776275758232 1loss/SRqqt3/assignment_loss‚1.507179212394274 /loss/FRt3/detection_loss‚2.9453526505402157 0loss/SRqqt2/detection_loss‚2.422592260156359 0loss/FRt4/assignment_loss‚1.2840974462234367 &loss/total_loss‚39.401779449799484 2loss/SRqqt4/assignment_loss‚0.9213841973262158 1loss/FBt3/assignment_loss‚0.04476982491902012 -loss/FBt1/detection_loss‚2.37011258168654 1loss/FBt1/assignment_loss‚0.16940663838514328 1loss/SRqqt3/detection_loss‚2.1891948626273208 1loss/SRqqt4/detection_loss‚0.7397018260179231 _step‚700 2loss/SRqqt1/assignment_loss‚1.0155630906644677 0loss/FBt2/assignment_loss‚0.1926749007357671 /loss/FRt4/detection_loss‚1.4506231070006803 /loss/FBt2/detection_loss‚2.4733268201351164 /loss/FBt4/detection_loss‚0.6043281335658275 trainer/global_step‚8499 % _timestamp‚1.7884970170457454e+09 0loss/FRt1/assignment_loss‚1.9814422480438068 /loss/FRt2/detection_loss‚2.5630666254587062 1loss/SRqqt1/detection_loss‚2.4091201722621918 _runtime‚9116.686030185 0loss/FRt2/assignment_loss‚2.4179009321857903 .loss/FRt1/detection_loss‚2.248240373826221 /loss/FRt3/assignment_loss‚3.273587681565966 1loss/FBt4/assignment_loss‚0.17570508918955213 .loss/FBt3/detection_loss‚2.532831147313118 epoch‚7!9‡aé#Û 1loss/FBt1/assignment_loss‚0.16940663838514328 1loss/SRqqt3/detection_loss‚2.1891948626273208 1loss/SRqqt4/detection_loss‚0.7397018260179231 _step‚700 2loss/SRqqt1/assignment_loss‚1.0155630906644677 0loss/FBt2/assignment_loss‚0.1926749007357671 /loss/FRt4/detection_loss‚1.4506231070006803 /loss/FBt2/detection_loss‚2.4733268201351164 /loss/FBt4/detection_loss‚0.6043281335658275 trainer/global_step‚8499 % _timestamp‚1.7884970170457454e+09 0loss/FRt1/assignment_loss‚1.9814422480438068 /loss/FRt2/detection_loss‚2.5630666254587062 1loss/SRqqt1/detection_loss‚2.4091201722621918 _runtime‚9116.686030185 0loss/FRt2/assignment_loss‚2.4179009321857903 .loss/FRt1/detection_loss‚2.248240373826221 /loss/FRt3/assignment_loss‚3.273587681565966 1loss/FBt4/assignment_loss‚0.17570508918955213 .loss/FBt3/detection_loss‚2.532831147313118 epoch‚7 2loss/SRqqt2/assignment_loss‚1.4695776275758232 1loss/SRqqt3/assignment_loss‚1.507179212394274 /loss/FRt3/detection_loss‚2.9453526505402157 0loss/SRqqt2/detection_loss‚2.422592260156359 0loss/FRt4/assignment_loss‚1.2840974462234367 &loss/total_loss‚39.401779449799484 2loss/SRqqt4/assignment_loss‚0.9213841973262158 1loss/FBt3/assignment_loss‚0.04476982491902012 -loss/FBt1/detection_loss‚2.37011258168654¼W\œê#‚(:‘ ü˜éÔ‚²ý® gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚172.72' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.544000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â/Åõƒë#‚(:ø ü˜éÔó¹·±! disk.nvme0n1.in‚ 29648.5703125) proc.memory.availableMB‚ 1679.04296875! proc.memory.rssMB‚ 2949.640625 network.recv‚ 3.021684e+06 disk.sda1.in‚0 disk.sda3.in‚0$ memory_percent‚89.75193500518799 cpu‚25.30466079165426( disk.nvme0c0n1.out‚217014.6123046875) disk./.usagePercent‚78.84825942935142 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.29296875 disk.dm-1.in‚0* proc.memory.percent‚18.003177642822266 proc.cpu.threads‚17% disk./.usageGB‚1174.8960456848145 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚217014.6123046875 disk.dm-0.in‚ 29648.5703125# disk.dm-0.out‚217014.6123046875 disk.sda3.out‚ 599.08984375 disk.dm-1.out‚ 599.08984375 network.sent‚ 2.1550719e+07 disk.sda.in‚0 disk.sda.out‚ 599.08984375 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 29648.5703125Õïêþì#‚(:„ ‹™éÔˆ««® gpu.0.gpu‚49.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.55' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.11 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úv3“…í#‚(:ú ‹™éÔ‹›”± disk.dm-1.out‚ 599.44921875 disk.drbd1260.out‚ 1.29296875 disk.sda.out‚ 599.44921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚29760.69140625 network.recv‚ 3.026709e+06) disk./.usagePercent‚78.84812400136582 disk.sda3.out‚ 599.44921875 disk.sda.in‚0& disk.nvme0n1.out‚217463.1396484375# disk.dm-0.out‚217463.1396484375$ memory_percent‚89.75310325622559* proc.memory.percent‚18.003177642822266 network.sent‚ 2.1658637e+07$ disk./.usageGB‚1174.894027709961 disk.sda3.in‚0 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚217463.1396484375" disk.nvme0n1.in‚29760.69140625 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-0.in‚29760.69140625( proc.memory.availableMB‚ 1678.8515625! proc.memory.rssMB‚ 2949.640625 cpu‚25.304329302699628™²U—’î#‚(:‡ š™éÔŸà­ gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.518' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5036 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þÜÓ€ï#‚(:õ š™éÔã–’° disk.sda.out‚ 599.8828125( proc.memory.availableMB‚ 1678.8515625" disk.nvme0n1.in‚29760.69140625 disk.dm-0.in‚29760.69140625 disk.dm-1.out‚ 599.8828125! proc.memory.rssMB‚ 2949.640625( disk.nvme0c0n1.out‚217719.6474609375 network.sent‚ 2.1665617e+07$ disk./.usageGB‚1174.894027709961 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 599.8828125 disk.sda.in‚0$ disk.nvme0c0n1.in‚29760.69140625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚1.296875$ memory_percent‚89.75310325622559* proc.memory.percent‚18.003177642822266 cpu‚25.304030167592032 proc.cpu.threads‚17& disk.nvme0n1.out‚217719.6474609375# disk.dm-0.out‚217719.6474609375 network.recv‚ 3.030088e+06) disk./.usagePercent‚78.84812400136582 disk.sda1.out‚0 disk.dm-1.in‚0÷xš{ð#‚(:’ ©™éÔ•õܯ gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚124.912' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.982400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü­”‚ñ#‚(:÷ ©™éÔñš©° network.sent‚ 2.1669939e+07# disk./.usageGB‚1174.89400100708 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 600.64453125" disk.nvme0n1.in‚29760.69140625 disk.dm-0.in‚29760.69140625$ memory_percent‚89.75157737731934 disk.sda3.in‚0 disk.drbd1260.out‚1.296875 disk.sda2.in‚0! proc.memory.rssMB‚ 2949.640625 proc.cpu.threads‚17 network.recv‚ 3.034862e+06 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚29760.69140625( disk.nvme0c0n1.out‚218023.9951171875* proc.memory.percent‚18.003177642822266 cpu‚25.303725195145002) disk./.usagePercent‚78.84812220931308 disk.sda3.out‚ 600.64453125 disk.dm-1.out‚ 600.64453125 disk.sda1.in‚0& disk.nvme0n1.out‚218023.9951171875# disk.dm-0.out‚218023.9951171875( proc.memory.availableMB‚ 1679.1015625ì¹H2ò#- _wandbruntime‚9167  _runtime‚9167‡ý™óÇó#Á & lr-AdamW/pg1‚0.0001896044464366925 & lr-AdamW/pg2‚0.0001896044464366925 trainer/global_step‚8549 % _timestamp‚1.7884970675507708e+09 _runtime‚9167.191242478 _step‚701šïqô#Æ % _timestamp‚1.788497ͱMÉ­0675507708e+09 _runtime‚9167.191242478 _step‚701 & lr-AdamW/pg1‚0.0001896044464366925 & lr-AdamW/pg2‚0.0001896044464366925 trainer/global_step‚8549½æ-ÓR2õ#- _wandbruntime‚9167  _runtime‚9167.ÒôÎü'ö#öO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚664964201527296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8766947692652 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚702 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚592305193519008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9175192822272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8766177044240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚607705628944896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8798620367872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚598906990574080 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150002422 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚583539033514128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47729496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150002422 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133745493 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 189865856 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚607705628944896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150002424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚674139394349568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚674139169975808 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚598907008577024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47728968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 trainer/global_step‚8549 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 189888872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9174992388608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚592305981206780 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 _runtime‚9167.192472492 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86015995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607704838297088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133744963 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884970675519698e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339868278 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚598906990574080 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚664964177587200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8797847723008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944†xž(÷#ûO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189888872 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚592305981206780 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 _runtime‚9167.192472492 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86015995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607704838297088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133744963 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884970675519698e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339868278 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚598906990574080 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚664964177587200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8797847723008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚664964201527296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8766947692652 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚702 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚592305193519008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9175192822272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8766177044240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚607705628944896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8798620367872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚598906990574080 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150002422 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚583539033514128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47729496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150002422 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133745493 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 189865856 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚607705628944896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150002424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚674139394349568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚674139169975808 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚598907008577024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47728968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 trainer/global_step‚8549 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 189888872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9174992388608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚665039549870080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8798852468736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133759778 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339929907 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189910609 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚583605082910528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189887567 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150019298 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9176242807296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚607774333643264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚598974690513408 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150019296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86025643 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚607773542982144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚607774333643264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚703 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚598974708516352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8798852468736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 339906863 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _runtime‚9168.084869446 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚592373048782626 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8767948832738 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47734658 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 $ _timestamp‚1.788497068443788e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚598974690513408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86025645 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150019296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚674215792677376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚598974708516352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚674215580899840Q¾ ¼ú#„N VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8767178184118 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚592373048782626 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8767948832738 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47734658 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 $ _timestamp‚1.788497068443788e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚598974690513408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.‡šNpeak‚ 1019674624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86025645 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150019296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚674215792677376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚598974708516352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚674215580899840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187837440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339929907 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150019298 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚665039549870080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8798852468736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133759778 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339929907 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189910609 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚583605082910528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189887567 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150019298 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9176242807296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚607774333643264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚598974690513408 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150019296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86025643 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚607773542982144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚607774333643264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚703 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚598974708516352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8798852468736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 339906863 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _runtime‚9168.084869446 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190954758 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86498395 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9227324099072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190977774 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚586842338262768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚611140086963712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚677958942765568 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150846124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190977774 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚595659373579888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150846122 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚611140877611520 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134489295 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚586842355302128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8848099423232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚668731594726400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341800880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47991433 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190954758 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8816247629348 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚8599 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150846122 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚595658585892116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86498397 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9227125762560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚611140086963712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134489830 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚602292005543424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8817018277760CM‰$ûO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚586842355302128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8848099423232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚668731594726400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341800880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47991433 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190954758 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8816247629348 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚8599 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150846122 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚595658585892116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86498397 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9227125762560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚611140086963712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134489830 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚602292005543424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8817018277760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚602292005543424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8848099423232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚611140877611520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47990900 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚602291987540480 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 341823898 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _step‚706 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8848872068096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190954758 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86498395 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9227324099072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190977774 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚586842338262768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚611140086963712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚677958942765568 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150846124 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190977774 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚595659373579888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150846122 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚611140877611520 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134489295 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046ÂÐp.Õ2Š$- _wandbruntime‚9218  _runtime‚9218,G÷Í'‹$€N VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8817248769226 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190976469 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚611209582309888 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9218.823565554 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8849876827136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚602359705482752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47996073 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190999511 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8849104168960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 341862509 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚611208791648768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚668806967009280 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚595726441155734 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341862509 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8849876827136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150862996 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9228373857792 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚595725653467754 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134504116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341839465 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190999511 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚602359705482752†` 'Œ$…N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134504655 _step‚707 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚595726441155734 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341862509 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8849876827136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150862996 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9228373857792 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚595725653467754 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134504116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341839465 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190999511 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚602359705482752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8817248769226 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190976469 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚611209582309888 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9218.823565554 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8849876827136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚602359705482752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47996073 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190999511 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8849104168960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 341862509 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚611208791648768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚668806967009280 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚614575335630336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9279566221312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151689824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚605677002509824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚681778578086912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚605676984506880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚605677002509824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48254808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚672499011865600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5901456.h¸¨Æ60050768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚614576126278144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151689822 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9279758266368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 _step‚710 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151689822 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135235073 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86980795 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚590145677090128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8899123768320 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135235605 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚672499035805696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚614576126278144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151689824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚599012765952996 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192066676 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192066676 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚8649 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86980797 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8898351123456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚605676984506880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 % _timestamp‚1.7884971656062276e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 343733482 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192043660 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚681778794072064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚599011978265224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚614575335630336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8898351123456 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8899123768320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192043660 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9279758266368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 _step‚710 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151689822 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135235073 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86980795 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚590145677090128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8899123768320 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135235605 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚672499035805696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚614576126278144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151689824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚599012765952996 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192066676 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192066676 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚8649 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86980797 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8898351123456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚605676984506880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 % _timestamp‚1.7884971656062276e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 343733482 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192043660 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚681778794072064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚599011978265224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚614575335630336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8898351123456 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8899123768320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192043660 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚614575335630336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9279566221312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151689824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚605677002509824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115ƺ4E2œ$- _wandbruntime‚9266  _runtime‚9266œp—E'$ÿM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192088413 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚590211726486528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 343795111 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151706696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151706698 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚605744702449152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚672574360208384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86990443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48260059 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8868090002954 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192065371 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135249969 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚614644830976512 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚605744702449152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚681855203816448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚599079045840862 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 343795111 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884971664783697e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8900128527360 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8867319354334 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135250504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 _runtime‚ 9266.11927221 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8899355869184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 343772067 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚614644040315392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192065371 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9280819667968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 179448832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚605744684446208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151706698 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚681855000427520 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚605744702449152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚681855203816448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚599079045840862 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 343795111 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884971664783697e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8900128527360 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8867319354334 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135250504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 _runtime‚ 9266.11927221 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8899355869184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 343772067 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚614644040315392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStaÔÄ?a+tsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192065371 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9280819667968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 179448832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚605744684446208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151706698 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚681855000427520 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 343795111 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151706696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151706698 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚605744702449152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚672574360208384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86990443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48260059 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8868090002954 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192065371 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135249969 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489ÇÎCÜJ2Ÿ$- _wandbruntime‚9266  _runtime‚9266¬P<Ø\ $Ö 0loss/FRt4/assignment_loss‚1.2785625002450414 /loss/FBt1/detection_loss‚2.7016426399350166 epoch‚7 % _timestamp‚1.7884971664806678e+09 0loss/FRt3/assignment_loss‚2.2049098972118264 0loss/SRqqt2/assignment_loss‚1.55665797040318 _runtime‚9266.120814266 0loss/FRt1/assignment_loss‚2.9603753497726037 /loss/FRt4/detection_loss‚1.3116336811540856 /loss/FBt4/detection_loss‚0.5621738072118855 1loss/SRqqt4/assignment_loss‚0.866479877296878 /loss/FRt3/detection_loss‚2.6648627483483516 0loss/SRqqt3/detection_loss‚1.508757756712536 /loss/FRt2/detection_loss‚2.8387561949804376 .loss/FBt2/detection_loss‚3.121163845062256 2loss/SRqqt3/assignment_loss‚1.3470737434820168 1loss/FBt2/assignment_loss‚0.11004020676708137 /loss/SRqqt2/detection_loss‚2.31103098934347 1loss/SRqqt4/detection_loss‚0.7212017647836401 _step‚712 0loss/FRt2/assignment_loss‚2.4858175781038074 2loss/SRqqt1/assignment_loss‚1.4716128846912673 1loss/FBt1/assignment_loss‚0.27370471564111654 %loss/total_loss‚39.92562495663152 trainer/global_step‚8649 1loss/FBt3/assignment_loss‚0.12513671255052827 1loss/FBt4/assignment_loss‚0.09292598226929136 /loss/FRt1/detection_loss‚1.9884122755182416 1loss/SRqqt1/detection_loss‚2.4776710949160834 /loss/FBt3/detection_loss‚2.9450207402308783\ÏçAa¡$Û epoch‚7 % _timestamp‚1.7884971664806678e+09 0loss/FRt3/assignment_loss‚2.2049098972118264 0loss/SRqqt2/assignment_loss‚1.55665797040318 _runtime‚9266.120814266 0loss/FRt1/assignment_loss‚2.9603753497726037 /loss/FRt4/detection_loss‚1.3116336811540856 /loss/FBt4/detection_loss‚0.5621738072118855 1loss/SRqqt4/assignment_loss‚0.866479877296878 /loss/FRt3/detection_loss‚2.6648627483483516 0loss/SRqqt3/detection_loss‚1.508757756712536 /loss/FRt2/detection_loss‚2.8387561949804376 .loss/FBt2/detection_loss‚3.121163845062256 2loss/SRqqt3/assignment_loss‚1.3470737434820168 1loss/FBt2/assignment_loss‚0.11004020676708137 /loss/SRqqt2/detection_loss‚2.31103098934347 1loss/SRqqt4/detection_loss‚0.7212017647836401 _step‚712 0loss/FRt2/assignment_loss‚2.4858175781038074 2loss/SRqqt1/assignment_loss‚1.4716128846912673 1loss/FBt1/assignment_loss‚0.27370471564111654 %loss/total_loss‚39.92562495663152 trainer/global_step‚8649 1loss/FBt3/assignment_loss‚0.12513671255052827 1loss/FBt4/assignment_loss‚0.09292598226929136 /loss/FRt1/detection_loss‚1.9884122755182416 1loss/SRqqt1/detection_loss‚2.4776710949160834 /loss/FBt3/detection_loss‚2.9450207402308783 0loss/FRt4/assignment_loss‚1.2785625002450414 /loss/FBt1/detection_loss‚2.7016426399350166È Öø¢$‚(:’ ’šéÔÜÎÖ¬ gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚118.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.729599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ +®ˆ£$‚(:ý ’šéÔÝåï¯ disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 2.191361e+07) disk./.usagePercent‚78.73214362809125 disk.sda3.in‚0 disk.sda3.out‚ 604.19921875 disk.drbd1260.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚30009.31640625" disk.nvme0n1.in‚30009.31640625' disk.nvme0n1.out‚220856.60791015625$ memory_percent‚93.16332340240479! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 proc.cpu.threads‚17 network.recv‚ 3.064768e+06% disk./.usageGB‚1173.1658363342285 disk.sda.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚220856.60791015625 disk.dm-0.in‚30009.31640625$ disk.dm-0.out‚220856.60791015625) proc.memory.availableMB‚ 1120.12109375 disk.dm-1.in‚0 disk.dm-1.out‚ 604.19921875 disk.drbd1260.out‚ 1.30859375 disk.sda.out‚ 604.19921875 cpu‚25.30138642100908õ⨟¤$‚(:’ ¡šéÔú…ù¬ gpu.0.gpu‚48.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.897' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.379399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d>µS†¥$‚(:û ¡šéÔ·Úή disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚30009.31640625 cpu‚25.301044225802535) disk./.usagePercent‚78.73214055600086 disk.sda.in‚0$ memory_percent‚93.16484928131104) proc.memory.availableMB‚ 1119.87109375 proc.cpu.threads‚17 network.recv‚ 3.071443e+06 disk.dm-1.out‚ 604.98828125 disk.drbd1260.out‚1.3125 disk.sda.out‚ 604.98828125 disk.sda1.in‚0$ disk.nvme0c0n1.in‚30009.31640625" disk.nvme0n1.in‚30009.31640625' disk.nvme0n1.out‚220952.19287109375% disk./.usageGB‚1173.1657905578613 disk.sda3.out‚ 604.98828125 disk.dm-1.in‚0) disk.nvme0c0n1.out‚220952.19287109375$ disk.dm-0.out‚220952.19287109375! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 disk.sda1.out‚0 network.sent‚ 2.2022802e+07 disk.sda3.in‚0 disk.drbd1260.in‚0Šq㎦$‚(:ƒ °šéÔ™ÊÛ¯ gpu.0.gpu‚44.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.3' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.86 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 {Õm§$‚(:â °šéÔ©„Ÿ´* proc.memory.percent‚18.003177642822266 network.recv‚ 3.075358e+06% disk./.usageGB‚1173.1632957458496 disk.sda2.in‚0$ disk.nvme0c0n1.in‚30009.31640625 disk.sda3.in‚0 disk.drbd1260.out‚1.3125 cpu‚25.300691331782684) disk./.usagePercent‚78.73197312707345 disk.sda1.out‚0" disk.nvme0n1.in‚30009.31640625 disk.nvme0n1.out‚ 221240.0625 disk.dm-1.out‚ 605.6328125! proc.memory.rssMB‚ 2949.640625 proc.cpu.threads‚17 network.sent‚ 2.2028685e+07 disk.sda1.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.out‚ 221240.0625 disk.drbd1260.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 1120.1328125 disk.dm-0.in‚30009.31640625 disk.dm-0.out‚ 221240.0625 disk.sda3.out‚ 605.6328125 disk.dm-1.in‚0 disk.sda.out‚ 605.6328125$ memory_percent‚93.16325187683105«Íkm2¨$- _wandbruntime‚9311  _runtime‚9311퉎È©$ trainer/global_step‚8699 $ _timestamp‚1.788497212187499e+09 _runtime‚9311.827934766 _step‚713 ' lr-AdamW/pg1‚0.00017516321686022546 ' lr-AdamW/pg2‚0.00017516321686022546½B™1ͪ$Ç _runtime‚9311.827934766 _step‚713 ' lr-AdamW/pg1‚0.00017516321686022546 ' lr-AdamW/pg2‚0.00017516321686022546 trainer/global_step‚8699 $ _timestamp‚1.788497212187499e+09Éߺò'2«$- _wandbruntime‚9311  _runtime‚9311åCïKû'¬$õO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152533522 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _step‚714 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚602365370638332 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚618011374944768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48519970 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8948602823680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135983167 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135982621 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚618010584296960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚676266429004800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8948602823680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚593448998878128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9332134912000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87463197 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚602366158326104 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8917159447976 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685598369774592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚618010584296960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884972121886024e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚618011374944768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 345666084 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193155578 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8949375468544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚609061999476224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9331940769792 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8916388799564 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚593448981838768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚609061999476224Û©}â?&­$úO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚618010584296960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚676266429004800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8948602823680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚593448998878128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9332134912000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87463197 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚602366158326104 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8917159447976 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685598369774592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚618010584296960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884972121886024e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚618011374944768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 345666084 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193155578 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8949375468544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚609061999476224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9331940769792 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8916388799564 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚593448981838768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚609061999476224 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152533522 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _step‚714 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚602365370638332 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚618011374944768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48519970 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8948602823680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135983167 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135982621 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚609129699415552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345727713 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚676341801287680 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚618080079643136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚618080079643136 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135997591 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 345704669 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚685674756338688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9333183605760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152550398 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345727713 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚685674984893440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8949607569408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8917389939442 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8950380227584 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8918160588062 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8699 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚618079288982016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48524748 $ _timestamp‚1.788497213025918e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345704669 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152550398 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152550396 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135998133 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚676341777347584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48525288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9332978991104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚609129699415552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚602432438213970 _step‚715 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075C÷õƒ '°$„N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9332978991104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚609129699415552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚602432438213970 _step‚715 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚609129699415552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345727713 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚676341801287680 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚618080079643136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚618080079643136 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135997591 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 345704669 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚685674756338688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9333183605760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152550398 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345727713 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚685674984893440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8949607569408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8917389939442 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8950380227584 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8918160588062 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8699 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚618079288982016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48524748 $ _timestamp‚1.788497213025918e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345704669 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152550398 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152550396 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135998133 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚676341777347584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48525288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328Ë«OËí2±$- _wandbruntime‚9312  _runtime‚9312öÖÉ|V²$Ð .loss/FRt4/detection_loss‚1.452961154117742 0loss/SRqqt2/detection_loss‚2.597341571535383 0loss/FRt1/assignment_loss‚2.5730453894493426 0loss/FRt4/assignment_loss‚1.1352930787731619 2loss/SRqqt1/assignment_loss‚1.5511480448501451 2loss/SRqqt4/assignment_loss‚0.8097069860383754 0loss/FBt1/assignment_loss‚0.3631962458871385 1loss/FBt3/assignment_loss‚0.21876766372219614 .loss/FRt1/detection_loss‚2.169098054078665 1loss/SRqqt3/detection_loss‚1.8860231858232748 .loss/FBt1/detection_loss‚2.476539358496666 /loss/FBt4/detection_loss‚0.6048597424209269 trainer/global_step‚8699 _runtime‚9312.668640186 _step‚716 1loss/SRqqt2/assignment_loss‚1.644605494503464 epoch‚7 % _timestamp‚1.7884972130284514e+09 0loss/FRt2/assignment_loss‚2.3824581558054145 .loss/FRt3/assignment_loss‚2.33333715482762 -loss/FRt3/detection_loss‚2.77705569486869 .loss/FBt2/detection_loss‚2.057564079761505 /loss/FBt3/detection_loss‚2.7044336994489035 2loss/SRqqt3/assignment_loss‚1.1969649590065945 0loss/FBt2/assignment_loss‚0.9120270717320719 1loss/FBt4/assignment_loss‚0.15732673823733445 0loss/SRqqt1/detection_loss‚2.159676675285612 1loss/SRqqt4/detection_loss‚0.8685757083894343 .loss/FRt2/detection_loss‚2.421213615434681 &loss/total_loss‚39.453219522494344(‰¤[³$Õ 0loss/FRt1/assignment_loss‚2.5730453894493426 0loss/FRt4/assignment_loss‚1.1352930787731619 2loss/SRqqt1/assignment_loss‚1.5511480448501451 2loss/SRqqt4/assignment_loss‚0.8097069860383754 0loss/FBt1/assignment_loss‚0.3631962458871385 1loss/FBt3/assignment_loss‚0.21876766372219614 .loss/FRt1/detection_loss‚2.169098054078665 1loss/SRqqt3/detection_loss‚1.8860231858232748 .loss/FBt1/detection_loss‚2.476539358496666 /loss/FBt4/detection_loss‚0.6048597424209269 trainer/global_step‚8699 _runtime‚9312.668640186 _step‚716 1loss/SRqqt2/assignment_loss‚1.644605494503464 epoch‚7 % _timestamp‚1.7884972130284514e+09 0loss/FRt2/assignment_loss‚2.3824581558054145 .loss/FRt3/assignment_loss‚2.33333715482762 -loss/FRt3/detection_loss‚2.77705569486869 .loss/FBt2/detection_loss‚2.057564079761505 /loss/FBt3/detection_loss‚2.7044336994489035 2loss/SRqqt3/assignment_loss‚1.1969649590065945 0loss/FBt2/assignment_loss‚0.9120270717320719 1loss/FBt4/assignment_loss‚0.15732673823733445 0loss/SRqqt1/detection_loss‚2.159676675285612 1loss/SRqqt4/detection_loss‚0.8685757083894343 .loss/FRt2/detection_loss‚2.421213615434681 &loss/total_loss‚39.453219522494344 .loss/FRt4/detection_loss‚1.452961154117742 0loss/SRqqt2/detection_loss‚2.597341571535383Ì}ì ß’´$‚(:‡ ¿šéÔ‹¨§® gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚165.356' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.0712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎÛ¾Ÿ€µ$‚(:õ ¿šéÔ‡¡¥² disk.sda.out‚ 605.99609375 disk.sda1.out‚0$ memory_percent‚93.16356182098389 network.sent‚ 2.2033849e+07$ disk./.usageGB‚1173.167251586914( disk.nvme0c0n1.out‚221336.1591796875& disk.nvme0n1.out‚221336.1591796875 disk.sda1.in‚0) proc.memory.availableMB‚ 1120.08203125* proc.memory.percent‚18.003177642822266 cpu‚25.300355079041186 disk.nvme0n1.in‚ 30141.421875 disk.sda3.out‚ 605.99609375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2949.640625" disk.nvme0c0n1.in‚ 30141.421875 disk.dm-0.in‚ 30141.421875# disk.dm-0.out‚221336.1591796875 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.31640625 proc.cpu.threads‚17 network.recv‚ 3.079212e+06) disk./.usagePercent‚78.73223860688647 disk.sda3.in‚0 disk.dm-1.out‚ 605.99609375 disk.sda.in‚0Æ~ý¡¶$‚(:’ ΚéÔÌäÙ¬ gpu.0.gpu‚28.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.056' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.611200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚044Ou·$‚(:ê ΚéԈ騮 disk.dm-0.in‚ 30141.421875! disk.dm-0.out‚221625.22265625 disk.sda3.in‚0$ memory_percent‚93.16506385803223 disk.sda2.in‚0 disk.nvme0n1.in‚ 30141.421875$ disk.nvme0n1.out‚221625.22265625( proc.memory.availableMB‚ 1119.8359375! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 proc.cpu.threads‚17( disk./.usagePercent‚78.7247987719339 disk.dm-1.in‚0 disk.dm-1.out‚ 606.3828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 606.3828125 network.sent‚ 2.214186e+07% disk./.usageGB‚1173.0563926696777 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 606.3828125" disk.nvme0c0n1.in‚ 30141.421875 cpu‚25.300004770135555 network.recv‚ 3.084633e+06 disk.drbd1260.out‚ 1.31640625 disk.sda2.out‚0& disk.nvme0c0n1.out‚221625.22265625 {uT¸$‚(:’ ÝšéÔ¦·»² gpu.0.gpu‚30.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.398' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.879599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹‚Rz¹$‚(:ï ÝšéÔÆ’¦´$ disk./.usageGB‚1173.232105255127 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚93.1635856628418! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 cpu‚25.29967965297467 disk.dm-1.out‚ 606.9609375 disk.drbd1260.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 30141.421875' disk.nvme0n1.out‚222003.16064453125 disk.dm-0.in‚ 30141.421875 disk.sda3.out‚ 606.9609375 network.sent‚ 2.2148224e+07 disk.drbd1260.out‚ 1.3203125 disk.sda.in‚0) disk.nvme0c0n1.out‚222003.16064453125 disk.nvme0n1.in‚ 30141.421875 disk.sda3.in‚0 network.recv‚ 3.08938e+06) disk./.usagePercent‚78.73659099097608 disk.sda.out‚ 606.9609375 disk.sda2.in‚0$ disk.dm-0.out‚222003.16064453125' proc.memory.availableMB‚ 1120.078125 proc.cpu.threads‚17vÄT’º$‚(:‡ ìšéÔ㉒´ gpu.0.gpu‚31.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.2062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö\L»$‚(:ô ìšéÔ³Âö" disk.nvme0c0n1.in‚ 30141.421875) disk.nvme0c0n1.out‚222356.52783203125# memory_percent‚93.1636095046997 network.recv‚ 3.093634e+06 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 607.52734375 disk.sda2.out‚0* proc.memory.percent‚18.003177642822266 cpu‚25.299429119696995$ disk./.usageGB‚1173.231761932373 disk.sda2.in‚0) proc.memory.availableMB‚ 1120.07421875! proc.memory.rssMB‚ 2949.640625 proc.cpu.threads‚17 network.sent‚ 2.2154232e+07' disk./.usagePercent‚78.736567950298 disk.dm-0.in‚ 30141.421875$ disk.dm-0.out‚222356.52783203125 disk.sda3.out‚ 607.52734375 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.3203125 disk.sda.in‚0 disk.nvme0n1.in‚ 30141.421875' disk.nvme0n1.out‚222356.52783203125 disk.dm-1.out‚ 607.52734375 disk.sda1.in‚0 disk.sda1.out‚0Æ·ƒ2¼$- _wandbruntime‚9361  _runtime‚9361DbaȽ$ % _timestamp‚1.7884972622711902e+09 _runtime‚ 9361.91155645 _step‚717 ' lr-AdamW/pg1‚0.00017044023995927218 ' lr-AdamW/pg2‚0.00017044023995927218 trainer/global_step‚8749 -`ŒÍ¾$Ç _step‚717 ' lr-AdamW/pg1‚0.00017044023995927218 ' lr-AdamW/pg2‚0.00017044023995927218 trainer/global_step‚8749 % _timestamp‚1.7884972622711902e+09 _runtime‚ 9361.91155645ͲÀÉ2¿$- _wandbruntime‚9361  _runtime‚9361QŠJÀ$öO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48783702 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48784237 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 % _timestamp‚1.7884972622723901e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153377222 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136729834 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚680033846144000 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚9361.912949004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621445832963584 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 194221464 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚612446996442624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚612446978439680 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚596752303626768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347598686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚612446996442624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153377224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8999627168768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87945597 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9384088982016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8967230033084 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194244480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚680033870084096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9384293609984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204627968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87945595 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194221464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8998854523904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚689417935126016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚596752320666128_<’Y(Á$ûO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48783702 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48784237 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 % _timestamp‚1.7884972622723901e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153377222 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136729834 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚680033846144000 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚9361.912949004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621445832963584 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 194221464 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚612446996442624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚612446978439680 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚596752303626768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347598686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚612446996442624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153377224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8999627168768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87945597 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9384088982016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8967230033084 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194244480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚680033870084096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9384293609984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204627968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87945595 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194221464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8998854523904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚689417935126016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚596752320666128α1)ü2Â$- _wandbruntime‚9362  _runtime‚9362ø¥Êd'Ã$€N D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9385329662464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194243175 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚612514678379008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚689494548089344 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347637271 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚680109194486784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8968231173170 trainer/global_step‚8749 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153394098 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153394096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8999859269632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347660315 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9385129242112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚719 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚596818370062528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136744150 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9362.992400094 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48788907 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153394096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚605785830587078 % _timestamp‚1.7884972633505356e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚621515328309760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87955243 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚605786618275058 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347637271 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136744695 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194266217 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚596818387101888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚621514537648640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194266217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚612514696381952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8999859269632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚680109218426880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489º0ÖòÄ$…N VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153394096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚605785830587078 % _timestamp‚1.7884972633505356e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚621515328309760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87955243 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚605786618275058 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347637271 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136744695 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194266217 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚596818387101888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚621514537648640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194266217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚612514696381952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8999859269632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚680109218426880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9385329662464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194243175 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚612514678379008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚689494548089344 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347637271 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚680109194486784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8968231173170 trainer/global_step‚8749 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153394098 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153394096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8999859269632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347660315 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9385129242112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚719 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚596818370062528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136744150 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9362.992400094 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48788907 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ÏJðCæ2Å$- _wandbruntime‚9362  _runtime‚9362±Ë`_Æ$Ù /loss/FRt3/detection_loss‚2.5707602959412794 1loss/SRqqt1/detection_loss‚2.9307510295639867 % _timestamp‚1.7884972633544745e+09 1loss/SRqqt4/assignment_loss‚0.786646648803178 .loss/FBt3/detection_loss‚2.205371013054481 epoch‚7 trainer/global_step‚8749 0loss/FRt2/assignment_loss‚2.6410064081991873 /loss/FRt1/detection_loss‚2.1180971235763737 1loss/FBt4/assignment_loss‚0.08620286989880638 1loss/SRqqt3/detection_loss‚2.3920341946424974 .loss/FRt4/assignment_loss‚0.96046175381967 2loss/FBt2/assignment_loss‚0.002379496128742176 0loss/FBt4/detection_loss‚0.42594204460536794 2loss/SRqqt1/assignment_loss‚1.4057696620407312 /loss/FRt2/detection_loss‚2.4642199690944007 1loss/SRqqt2/detection_loss‚2.1705645322799683 1loss/SRqqt4/detection_loss‚0.5831876888359878 0loss/FRt3/assignment_loss‚2.5228580756065173 2loss/SRqqt3/assignment_loss‚1.3231974748834487 .loss/FBt2/detection_loss‚3.561396598815918 _step‚720 2loss/SRqqt2/assignment_loss‚1.4424325563678784 1loss/FBt3/assignment_loss‚0.23828527150163398 .loss/FRt4/detection_loss‚1.477851844525763 /loss/FBt1/detection_loss‚2.8363285288214684 %loss/total_loss‚40.30714209649189 _runtime‚9362.994786971 0loss/FRt1/assignment_loss‚3.0819555317483296 1loss/FBt1/assignment_loss‚0.07944148373627513’WÒJdÇ$Þ .loss/FBt2/detection_loss‚3.561396598815918 _step‚720 2loss/SRqqt2/assignment_loss‚1.4424325563678784 1loss/FBt3/assignment_loss‚0.23828527150163398 .loss/FRt4/detection_loss‚1.477851844525763 /loss/FBt1/detection_loss‚2.8363285288214684 %loss/total_loss‚40.30714209649189 _runtime‚9362.994786971 0loss/FRt1/assignment_loss‚3.0819555317483296 1loss/FBt1/assignment_loss‚0.07944148373627513 /loss/FRt3/detection_loss‚2.5707602959412794 1loss/SRqqt1/detection_loss‚2.9307510295639867 % _timestamp‚1.7884972633544745e+09 1loss/SRqqt4/assignment_loss‚0.786646648803178 .loss/FBt3/detection_loss‚2.205371013054481 epoch‚7 trainer/global_step‚8749 0loss/FRt2/assignment_loss‚2.6410064081991873 /loss/FRt1/detection_loss‚2.1180971235763737 1loss/FBt4/assignment_loss‚0.08620286989880638 1loss/SRqqt3/detection_loss‚2.3920341946424974 .loss/FRt4/assignment_loss‚0.96046175381967 2loss/FBt2/assignment_loss‚0.002379496128742176 0loss/FBt4/detection_loss‚0.42594204460536794 2loss/SRqqt1/assignment_loss‚1.4057696620407312 /loss/FRt2/detection_loss‚2.4642199690944007 1loss/SRqqt2/detection_loss‚2.1705645322799683 1loss/SRqqt4/detection_loss‚0.5831876888359878 0loss/FRt3/assignment_loss‚2.5228580756065173 2loss/SRqqt3/assignment_loss‚1.3231974748834487Ð?&ÈÈ$‚(:’ ûšéԮѾ® gpu.0.gpu‚49.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.895' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.379000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´úªÔÉ$‚(:ö ûšéÔòï³± disk.drbd1260.out‚ 1.32421875! disk.nvme0n1.in‚ 30298.9609375$ memory_percent‚93.16549301147461* proc.memory.percent‚18.003177642822266 proc.cpu.threads‚17) disk./.usagePercent‚78.72531718719074 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 608.00390625# disk.dm-0.out‚222622.3544921875! proc.memory.rssMB‚ 2949.640625 cpu‚25.29914148183167 network.sent‚ 2.2172275e+07 disk.sda3.out‚ 608.00390625 disk.dm-1.in‚0 disk.dm-1.out‚ 608.00390625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 network.recv‚ 3.097102e+06% disk./.usageGB‚1173.0641174316406 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 30298.9609375( disk.nvme0c0n1.out‚222622.3544921875& disk.nvme0n1.out‚222622.3544921875 disk.dm-0.in‚ 30298.9609375' proc.memory.availableMB‚ 1119.765625@JáJÊ$‚(:’ Š›éÔÎÉÊ® gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚127.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.568199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®æ¼xË$‚(:í Š›éÔ½ü° disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 30298.9609375 disk.dm-1.out‚ 608.40625 disk.sda.out‚ 608.40625 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚93.16244125366211! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 network.sent‚ 2.2269189e+07& disk.nvme0n1.out‚222747.0888671875 disk.sda2.in‚0' proc.memory.availableMB‚ 1120.265625 proc.cpu.threads‚17) disk./.usagePercent‚78.72085830396688% disk./.usageGB‚1172.9976768493652! disk.nvme0n1.in‚ 30298.9609375 disk.sda3.out‚ 608.40625 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.32421875 cpu‚25.298828444878037 network.recv‚ 3.10288e+06( disk.nvme0c0n1.out‚222747.0888671875 disk.dm-0.in‚ 30298.9609375# disk.dm-0.out‚222747.0888671875h¦)ÓÌ$‚(:’ ™›éÔµ­ gpu.0.gpu‚41.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.592' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.918400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v·õYvÍ$‚(:ë ™›éÔ›…¾° disk.sda3.out‚ 608.84375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 30298.9609375) disk./.usagePercent‚78.72916600446051 disk.sda.out‚ 608.84375$ memory_percent‚93.16396713256836* proc.memory.percent‚18.003177642822266 cpu‚25.298490390945044 proc.cpu.threads‚17 disk.drbd1260.out‚1.328125( disk.nvme0c0n1.out‚222969.4833984375! disk.nvme0n1.in‚ 30298.9609375 disk.dm-0.in‚ 30298.9609375' proc.memory.availableMB‚ 1120.015625! proc.memory.rssMB‚ 2949.640625 network.sent‚ 2.2275493e+07 disk.sda3.in‚0 disk.dm-1.out‚ 608.84375 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚222969.4833984375# disk.dm-0.out‚222969.4833984375 network.recv‚ 3.107723e+06$ disk./.usageGB‚1173.121467590332U"lh2Î$- _wandbruntime‚9409  _runtime‚9409|jjŒÈÏ$ ' lr-AdamW/pg1‚0.00016576429460173284 ' lr-AdamW/pg2‚0.00016576429460173284 trainer/global_step‚8799 % _timestamp‚1.7884973099355469e+09 _runtime‚ 9409.57593438 _step‚721½0ï ÍÐ$Ç _runtime‚ 9409.57593438 _step‚721 ' lr-AdamW/pg1‚0.00016576429460173284 ' lr-AdamW/pg2‚0.00016576429460173284 trainer/global_step‚8799 % _timestamp‚1.7884973099355469e+09ÑÍR^K2Ñ$- _wandbruntime‚9409  _runtime‚94097¥žÀü'Ò$öO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚683801263283200 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154220922 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9049106224128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137474922 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9016529969780 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195310366 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9049106224128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 349531288 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49046927 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9436299893248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9049878868992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9436494035456 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195333382 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚615831993409024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚8799 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624881872278016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚615831993409024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137475473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884973099368916e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615831975406080 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154220924 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195333382 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚609072155384548 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615831975406080 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚722 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚600055625414768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624881081630208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88427995 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9017300618192 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154220924 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9049878868992 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚683801287223296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚600055642454128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349554306 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚693237781258752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚693237563176448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚624881872278016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154220922 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88427997 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624881081630208 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚609072943072320 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615831975406080 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154220924 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195333382 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚609072155384548 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615831975406080 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚722 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚600055625414768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624881081630208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88427995 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9017300618192 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154220924 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9049878868992 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚683801287223296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚600055642454128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349554306 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚693237781258752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚693237563176448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚624881872278016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154220922 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88427997 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624881081630208 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚609072943072320 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚8799 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624881872278016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚615831993409024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137475473 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884973099368916e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137489913 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9050883628032 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚Å/ÿŒ$23042 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9437328685568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚693313940311552 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154237798 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624950576976384 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88437645 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚683876611625984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚624950576976384 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349592917 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9017531109658 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88437643 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154237796 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349569873 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9050110969856 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154237798 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49052270 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚600121708889888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚609140010648166 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624949786315264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49052814 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195355119 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9050883628032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195355119 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236943360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 213003264 _runtime‚9410.595003596 _step‚723 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349592917 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884973109538658e+09G‹~Ä 'Ö$…N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624950576976384 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88437645 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚683876611625984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚624950576976384 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349592917 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9017531109658 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88437643 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154237796 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349569873 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9050110969856 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154237798 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49052270 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚600121708889888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚609140010648166 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624949786315264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49052814 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195355119 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9050883628032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195355119 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236943360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 213003264 _runtime‚9410.595003596 _step‚723 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349592917 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884973109538658e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349569873 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137489913 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9050883628032 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9437328685568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚693313940311552 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154237798 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944Óª[ÍÙ2×$- _wandbruntime‚9410  _runtime‚9410Öï{OZØ$Ô 1loss/SRqqt2/detection_loss‚2.5521412367622056 epoch‚7 $ _timestamp‚1.788497310956988e+09 2loss/SRqqt4/assignment_loss‚0.8190340949751606 /loss/FRt1/detection_loss‚2.2523703068131353 /loss/FRt4/detection_loss‚1.4990425674965786 /loss/FBt4/detection_loss‚0.7546369018132931 1loss/FBt2/assignment_loss‚0.20992306645416647 /loss/FRt3/detection_loss‚2.8617462697956295 .loss/FRt1/assignment_loss‚3.03534706336696 0loss/FBt3/assignment_loss‚0.1392778633744456 1loss/SRqqt4/detection_loss‚0.9424544100818987 /loss/FBt1/detection_loss‚2.1083339717653065 2loss/SRqqt2/assignment_loss‚1.5820309209326904 .loss/FBt2/detection_loss‚1.858175027370453 .loss/FBt3/detection_loss‚2.843270242214203 %loss/total_loss‚40.24679037821274 trainer/global_step‚8799 _step‚724 0loss/FRt2/assignment_loss‚3.2655187652957056 0loss/FRt3/assignment_loss‚2.2548121213912964 0loss/FBt1/assignment_loss‚0.4427173477411088 1loss/FBt4/assignment_loss‚0.08686277883316326 1loss/SRqqt3/detection_loss‚1.9170955841739972 _runtime‚9410.597196758 /loss/FRt4/assignment_loss‚1.284955543856467 2loss/SRqqt1/assignment_loss‚1.4302457295358182 2loss/SRqqt3/assignment_loss‚1.6989648580063312 /loss/FRt2/detection_loss‚2.3207950622916895 0loss/SRqqt1/detection_loss‚2.087038643871035û5_è_Ù$Ù .loss/FRt1/assignment_loss‚3.03534706336696 0loss/FBt3/assignment_loss‚0.1392778633744456 1loss/SRqqt4/detection_loss‚0.9424544100818987 /loss/FBt1/detection_loss‚2.1083339717653065 2loss/SRqqt2/assignment_loss‚1.5820309209326904 .loss/FBt2/detection_loss‚1.858175027370453 .loss/FBt3/detection_loss‚2.843270242214203 %loss/total_loss‚40.24679037821274 trainer/global_step‚8799 _step‚724 0loss/FRt2/assignment_loss‚3.2655187652957056 0loss/FRt3/assignment_loss‚2.2548121213912964 0loss/FBt1/assignment_loss‚0.4427173477411088 1loss/FBt4/assignment_loss‚0.08686277883316326 1loss/SRqqt3/detection_loss‚1.9170955841739972 _runtime‚9410.597196758 /loss/FRt4/assignment_loss‚1.284955543856467 2loss/SRqqt1/assignment_loss‚1.4302457295358182 2loss/SRqqt3/assignment_loss‚1.6989648580063312 /loss/FRt2/detection_loss‚2.3207950622916895 0loss/SRqqt1/detection_loss‚2.087038643871035 1loss/SRqqt2/detection_loss‚2.5521412367622056 epoch‚7 $ _timestamp‚1.788497310956988e+09 2loss/SRqqt4/assignment_loss‚0.8190340949751606 /loss/FRt1/detection_loss‚2.2523703068131353 /loss/FRt4/detection_loss‚1.4990425674965786 /loss/FBt4/detection_loss‚0.7546369018132931 1loss/FBt2/assignment_loss‚0.20992306645416647 /loss/FRt3/detection_loss‚2.8617462697956295Ô‹ùíÚ$‚(:’ ¨›éÔ±€° gpu.0.gpu‚32.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.596' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.119200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kƒƒºÛ$‚(:ö ¨›éÔÙêƳ disk.drbd1260.in‚0 disk.drbd1260.out‚1.328125 disk.sda3.in‚0 disk.dm-1.out‚ 609.57421875 disk.sda.out‚ 609.57421875 disk.sda1.out‚0$ memory_percent‚93.16418170928955) proc.memory.availableMB‚ 1119.98046875! proc.memory.rssMB‚ 2949.640625 disk.sda3.out‚ 609.57421875 disk.dm-1.in‚0% disk./.usageGB‚1173.3084678649902! disk.nvme0n1.in‚ 30298.9609375 disk.dm-0.in‚ 30298.9609375 disk.sda2.out‚0 cpu‚25.298160173449716 proc.cpu.threads‚17( disk.nvme0c0n1.out‚223377.3232421875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚18.003177642822266# disk.dm-0.out‚223377.3232421875 network.sent‚ 2.2292477e+07 network.recv‚ 3.111674e+06( disk./.usagePercent‚78.7417157497965# disk.nvme0c0n1.in‚ 30298.9609375& disk.nvme0n1.out‚223377.3232421875<Ó%lÜ$‚(:’ ·›éÔ”¸¼® gpu.0.gpu‚29.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.594' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.918799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î¢\„Ý$‚(:ù ·›éÔƟñ network.recv‚ 3.117362e+06) disk./.usagePercent‚78.74166352425952 disk.sda3.in‚0 disk.dm-1.out‚ 610.30078125 disk.drbd1260.out‚ 1.33203125( disk.nvme0c0n1.out‚223855.6591796875& disk.nvme0n1.out‚223855.6591796875 disk.dm-0.in‚30399.01171875 network.sent‚ 2.2387558e+07 disk.drbd1260.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚30399.01171875$ memory_percent‚93.16263198852539' proc.memory.availableMB‚ 1120.234375! proc.memory.rssMB‚ 2949.640625* proc.memory.percent‚18.003177642822266 disk.sda.out‚ 610.30078125 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚1173.307689666748 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚30399.01171875 disk.sda3.out‚ 610.30078125# disk.dm-0.out‚223855.6591796875 cpu‚25.297885183522027 proc.cpu.threads‚17ÐuÏ’Þ$‚(:‡ Æ›éÔÌ¡Ô± gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úNcŒƒß$‚(:ø Æ›éÔÚ‰ú´! disk.nvme0n1.in‚ 30417.5703125& disk.nvme0n1.out‚224111.9287109375# disk.dm-0.out‚224111.9287109375! proc.memory.rssMB‚ 2949.640625 cpu‚25.297607156779527 proc.cpu.threads‚17 network.sent‚ 2.2392653e+07) disk./.usagePercent‚78.74585795170051$ disk./.usageGB‚1173.370189666748 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 30417.5703125$ memory_percent‚93.16418170928955 network.recv‚ 3.120909e+06 disk.dm-1.out‚ 610.72265625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.33203125 disk.sda.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.003177642822266 disk.sda3.out‚ 610.72265625 disk.sda.out‚ 610.72265625 disk.sda1.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚224111.9287109375 disk.dm-0.in‚ 30417.5703125) proc.memory.availableMB‚ 1119.98046875¨ÿp³2à$- _wandbruntime‚9462  _runtime‚9462²6šÉá$à ' lr-AdamW/pg1‚0.00016113635524761148 ' lr-AdamW/pg2‚0.00016113635524761148 trainer/global_step‚8849 % _timestamp‚1.7884973630867715e+09 _runtime‚9462.727198645 _step‚725dŽÎPÎâ$È % _timestamp‚1.7884973630867715e+09 _runtime‚9462.727198645 _step‚725 ' lr-AdamW/pg1‚0.00016113635524761148 ' lr-AdamW/pg2‚0.00016113635524761148 trainer/global_step‚8849Õ܈éù2ã$- _wandbruntime‚9462  _runtime‚9462uB¸Kä$öO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚619216972372480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9488620947968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚612425547757656 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49311164 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚628317120944640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155064624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196422284 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚628316330296832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88910397 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9488817187328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚697057301370368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚619216990375424 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155064622 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚697057521549824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚687568680422400 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚628316330296832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 351463890 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138221026 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9067371203300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚628317120944640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138221561 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9100130569216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚612426335445428 _runtime‚9462.728519203 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9100130569216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9099357924352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 trainer/global_step‚8849 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚619216972372480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 351486908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚687568704362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚603358964242128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49310631 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9099357924352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0‘‡(å$ûO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9099357924352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 trainer/global_step‚8849 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚619216972372480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 351486908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚687568704362496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚603358964242128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49310631 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9099357924352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚619216972372480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9488620947968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚612425547757656 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49311164 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚628317120944640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155064624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196422284 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚628316330296832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88910397 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9488817187328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚697057301370368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚619216990375424 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155064622 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚697057521549824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚687568680422400 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚628316330296832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 351463890 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138221026 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9067371203300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚628317120944640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138221561 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9100130569216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚612426335445428 _runtime‚9462.728519203 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9100130569216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846Ö"42æ$- _wandbruntime‚9463  _runtime‚9463ÐÁ–'ç$€N EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138235911 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚697133922186752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49316398 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚697133704117760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9489675352576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚8849 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9101135328256 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196420979 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612493403021274 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155081498 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196444021 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9489869481472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚612492615333294 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9068372343386 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚628385825643008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603425013638528 _step‚727 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196420979 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351525519 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49315868 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603425030677888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 _runtime‚9463.780614615 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138235911 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚697133922186752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49316398 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚697133704117760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9489675352576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚8849 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9101135328256 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196420979 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612493403021274 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155081498 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196444021 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9489869481472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚612492615333294 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9068372343386 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚628385825643008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603425013638528 _step‚727 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196420979 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351525519 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49315868 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603425030677888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 _runtime‚9463.780614615 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚700877104948736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚615778940130764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚691336121501696 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138967961 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197511186 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197488170 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9540783013376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 197511186 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚622601969338880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 % _timestamp‚1.7884974142483487e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353396492 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9150382269440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚691336097561600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9116671139996 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9149609624576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9540983447040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155908322 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚622601969338880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚700876880574976 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚631751578963456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚631751578963456§$(ù$ûO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚615778940130764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚691336121501696 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138967961 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197511186 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197488170 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9540783013376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 197511186 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚622601969338880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 % _timestamp‚1.7884974142483487e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353396492 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9150382269440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚691336097561600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9116671139996 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9149609624576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9540983447040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155908322 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚622601969338880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚700876880574976 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚631751578963456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚631751578963456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138968518 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚631752369611264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚700877104948736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Ú$0å2ú$- _wandbruntime‚9514  _runtime‚9514ÃQRÜû$ÿM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155925196 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 197532923 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚622669687281152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚622669687281152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49580498 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚622669669278208 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155925198 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9541815463936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_tr¥C¾š)ain_batch_end/active_bytes.small_pool.current‚ 772658176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89402445 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 353435077 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚691411469844480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚615846007706402 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9150614370304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚631821074309632 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788497415166501e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197532923 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138982941 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚731 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9150614370304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9151387028480 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚691411445904384 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353458121 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚615846795394382 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 trainer/global_step‚8899L…¸í 'ü$„N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155925196 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 197532923 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚622669687281152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚622669687281152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49580498 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚622669669278208 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155925198 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9541815463936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89402445 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 353435077 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚691411469844480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚615846007706402 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9150614370304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚631821074309632 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788497415166501e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197532923 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138982941 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚731 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9150614370304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9151387028480 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚691411445904384 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353458121 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚615846795394382 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 trainer/global_step‚8899 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115Ûßñ2ý$- _wandbruntime‚9514  _runtime‚9514Çãœ?\þ$Ö 0loss/FRt3/assignment_loss‚2.5734182397524514 2loss/SRqqt2/assignment_loss‚1.5242366677728192 1loss/SRqqt4/assignment_loss‚0.942684406656082 1loss/FBt1/assignment_loss‚0.01545346602621862 1loss/FBt4/assignment_loss‚0.12071442888496292 1loss/SRqqt4/detection_loss‚0.7456485491108011 %loss/total_loss‚41.77811519923867 trainer/global_step‚8899 .loss/FRt3/detection_loss‚3.143817356654576 /loss/FRt1/detection_loss‚2.3126786122488423 0loss/SRqqt3/detection_loss‚1.792927159089595 epoch‚7 0loss/FRt1/assignment_loss‚2.9951135141904963 2loss/SRqqt1/assignment_loss‚1.1506009285982985 2loss/FBt3/assignment_loss‚0.015599895122779373 $ _timestamp‚1.788497415168889e+09 3loss/FBt2/assignment_loss‚0.0021290724172205147 1loss/SRqqt1/detection_loss‚2.6642481913933387 .loss/FBt2/detection_loss‚2.478566381666395 2loss/SRqqt3/assignment_loss‚1.3658594023901969 /loss/FRt2/detection_loss‚2.5140708709296824 /loss/FRt4/detection_loss‚1.4726022665272467 _runtime‚ 9514.80908687 .loss/FRt2/assignment_loss‚3.28108452139674 /loss/FBt3/detection_loss‚3.2394389510154724 /loss/FRt4/assignment_loss‚1.060783525928855 1loss/SRqqt2/detection_loss‚2.2691830150012313 .loss/FBt1/detection_loss‚3.690048325061798 /loss/FBt4/detection_loss‚0.4072074514025679 _step‚732iüéaÿ$Û 3loss/FBt2/assignment_loss‚0.0021290724172205147 1loss/SRqqt1/detection_loss‚2.6642481913933387 .loss/FBt2/detection_loss‚2.478566381666395 2loss/SRqqt3/assignment_loss‚1.3658594023901969 /loss/FRt2/detection_loss‚2.5140708709296824 /loss/FRt4/detection_loss‚1.4726022665272467 _runtime‚ 9514.80908687 .loss/FRt2/assignment_loss‚3.28108452139674 /loss/FBt3/detection_loss‚3.2394389510154724 /loss/FRt4/assignment_loss‚1.060783525928855 1loss/SRqqt2/detection_loss‚2.2691830150012313 .loss/FBt1/detection_loss‚3.690048325061798 /loss/FBt4/detection_loss‚0.4072074514025679 _step‚732 0loss/FRt3/assignment_loss‚2.5734182397524514 2loss/SRqqt2/assignment_loss‚1.5242366677728192 1loss/SRqqt4/assignment_loss‚0.942684406656082 1loss/FBt1/assignment_loss‚0.01545346602621862 1loss/FBt4/assignment_loss‚0.12071442888496292 1loss/SRqqt4/detection_loss‚0.7456485491108011 %loss/total_loss‚41.77811519923867 trainer/global_step‚8899 .loss/FRt3/detection_loss‚3.143817356654576 /loss/FRt1/detection_loss‚2.3126786122488423 0loss/SRqqt3/detection_loss‚1.792927159089595 epoch‚7 0loss/FRt1/assignment_loss‚2.9951135141904963 2loss/SRqqt1/assignment_loss‚1.1506009285982985 2loss/FBt3/assignment_loss‚0.015599895122779373 $ _timestamp‚1.788497415168889e+09Üš¿9—€%‚(:’ ‘œéÔ͖ɱ gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.432' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.486399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûádgk%‚(:à ‘œéÔ°½û´$ memory_percent‚93.16480159759521# proc.memory.rssMB‚ 2949.64453125 network.sent‚ 2.2524255e+07 network.recv‚ 3.143966e+06) disk./.usagePercent‚78.78836365063303 disk.dm-1.out‚613.375 cpu‚25.296116402858043 disk.sda3.in‚0 disk.sda3.out‚613.375 disk.drbd1260.out‚ 1.34765625 disk.sda.in‚0 disk.sda1.out‚0! disk.nvme0n1.out‚ 225854.84375 disk.dm-0.in‚30530.57421875 disk.dm-0.out‚ 225854.84375% disk./.usageGB‚1174.0035552978516 disk.drbd1260.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚30530.57421875) proc.memory.availableMB‚ 1119.87890625* proc.memory.percent‚18.003201484680176 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda.out‚613.375 disk.sda1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.out‚ 225854.84375" disk.nvme0n1.in‚30530.57421875ð-­‚%‚(:’  œéÔÕÝ»° gpu.0.gpu‚47.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.384' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.876799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ÂßÞ|ƒ%‚(:ñ  œéÔë³é² disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0c0n1.out‚226207.2265625 disk.dm-0.out‚226207.2265625* proc.memory.percent‚18.003201484680176 network.sent‚ 2.2632833e+07 disk.drbd1260.out‚ 1.34765625 disk.sda1.in‚0$ disk.nvme0c0n1.in‚30530.57421875$ memory_percent‚93.16627979278564 proc.cpu.threads‚17 network.recv‚ 3.149216e+06( disk./.usagePercent‚78.7883290896159 disk.dm-1.out‚ 613.8984375 disk.drbd1260.in‚0 disk.sda.out‚ 613.8984375" disk.nvme0n1.in‚30530.57421875 cpu‚25.295822339601923 disk.sda3.in‚0 disk.sda3.out‚ 613.8984375 disk.dm-1.in‚0# disk.nvme0n1.out‚226207.2265625 disk.dm-0.in‚30530.57421875) proc.memory.availableMB‚ 1119.63671875# proc.memory.rssMB‚ 2949.64453125% disk./.usageGB‚1174.0030403137207 disk.sda.in‚0 disk.sda1.out‚0æy‘„%‚(:† ¯œéÔøáá® gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚127.745' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.549 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªN‹…%‚(:€ ¯œéԴ┳ disk.sda1.out‚0' disk.nvme0n1.out‚226463.77392578125$ disk.dm-0.out‚226463.77392578125$ memory_percent‚93.16647052764893% disk./.usageGB‚1174.7258796691895 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚30530.57421875) proc.memory.availableMB‚ 1119.60546875 cpu‚25.295515625904713 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.3515625" disk.nvme0n1.in‚30530.57421875# proc.memory.rssMB‚ 2949.64453125* proc.memory.percent‚18.003201484680176 proc.cpu.threads‚17 network.sent‚ 2.2637223e+07) disk./.usagePercent‚78.83683944526403 disk.sda3.in‚0 disk.sda3.out‚ 614.19921875 disk.dm-1.in‚0 disk.sda.in‚0) disk.nvme0c0n1.out‚226463.77392578125 disk.dm-0.in‚30530.57421875 network.recv‚ 3.152574e+06 disk.dm-1.out‚ 614.19921875 disk.sda.out‚ 614.19921875 disk.sda1.in‚0ç6{è2†%- _wandbruntime‚9562  _runtime‚9562©Øùɇ%à ' lr-AdamW/pg2‚0.00015202834216675211 trainer/global_step‚8949 % _timestamp‚1.7884974628162882e+09 _runtime‚9562.456949334 _step‚733 ' lr-AdamW/pg1‚0.00015202834216675211ïR»PΈ%È % _timestamp‚1.7884974628162882e+09 _runtime‚9562.456949334 _step‚733 ' lr-AdamW/pg1‚0.00015202834216675211 ' lr-AdamW/pg2‚0.00015202834216675211 trainer/global_step‚8949ÝšöS2‰%- _wandbruntime‚9562  _runtime‚9562È‹ú96Š%öO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9200633969664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198600088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚635186827630080 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198577072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9166741725104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚625986966305280 _runtime‚9562.458151942 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚625986984308224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚625986984308224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89875195 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139714646 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49839451 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚609965590778768 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156752022 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156752024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚635187618277888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚635187618277888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚635186827630080 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198577072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9166741725104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚625986966305280 _runtime‚9562.458151942 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚625986984308224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚625986984308224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89875195 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139714646 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49839451 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚609965590778768 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156752022 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156752024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚635187618277888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚635187618277888 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156768896 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198621825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49844806 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚610031657214528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156768898 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚635256322976256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89884845 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9200866070528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355367679 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355390723 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49845343 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9201638728704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139730188 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚704773409268736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚619199400079510 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚695178863043584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9594309281792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚635255532315136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236943360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 trainer/global_step‚8949 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚626054684247552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156768898 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚735 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9167742864982 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139729649 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9594522285056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1š34Þ%Ž%…N YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9201638728704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139730188 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚704773409268736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚619199400079510 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚695178863043584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9594309281792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚635255532315136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236943360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 trainer/global_step‚8949 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚626054684247552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156768898 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 _step‚735 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9167742864982 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139729649 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9594522285056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚635256322976256 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚626054684247552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚9563.420445978 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156768896 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198621825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49844806 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚610031657214528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156768898 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚635256322976256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89884845 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9200866070528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355367679 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355390723 F8DeviceStaxÿê-tsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49845343 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825ß:¤Ô2%- _wandbruntime‚9563  _runtime‚9563wÓÅ][%Õ .loss/FRt3/detection_loss‚2.580568592995405 /loss/FBt2/detection_loss‚2.2523875615813513 %loss/total_loss‚40.33835871314365 % _timestamp‚1.7884974637829654e+09 2loss/SRqqt2/assignment_loss‚1.4531628263719154 .loss/FRt4/detection_loss‚1.341972102019666 /loss/FBt3/detection_loss‚3.7518510937690737 /loss/FBt4/detection_loss‚0.3775228271439958 _step‚736 0loss/FRt1/assignment_loss‚2.7259705116351447 /loss/FRt4/assignment_loss‚1.243328775184742 2loss/SRqqt4/assignment_loss‚0.7456171344677659 0loss/FBt1/assignment_loss‚0.3568819748208106 1loss/FBt4/assignment_loss‚0.19645873002626757 1loss/SRqqt1/detection_loss‚2.3105824487283826 trainer/global_step‚8949 _runtime‚9563.423380941 0loss/FRt2/assignment_loss‚3.4224554364745683 /loss/FRt3/assignment_loss‚2.723410854116082 2loss/SRqqt1/assignment_loss‚1.0011370098218322 2loss/SRqqt3/assignment_loss‚1.5860904151728998 /loss/FRt2/detection_loss‚2.5022578448668207 0loss/SRqqt2/detection_loss‚2.398653586705526 1loss/SRqqt4/detection_loss‚0.7569742774994103 epoch‚7 0loss/FBt3/assignment_loss‚0.1108566862675616 .loss/FRt1/detection_loss‚2.189058294907833 1loss/FBt2/assignment_loss‚0.41271306761208587 1loss/SRqqt3/detection_loss‚1.8141830679960549 /loss/FBt1/detection_loss‚2.0842635929584503Vq¦`‘%Ú 2loss/SRqqt3/assignment_loss‚1.5860904151728998 /loss/FRt2/detection_loss‚2.5022578448668207 0loss/SRqqt2/detection_loss‚2.398653586705526 1loss/SRqqt4/detection_loss‚0.7569742774994103 epoch‚7 0loss/FBt3/assignment_loss‚0.1108566862675616 .loss/FRt1/detection_loss‚2.189058294907833 1loss/FBt2/assignment_loss‚0.41271306761208587 1loss/SRqqt3/detection_loss‚1.8141830679960549 /loss/FBt1/detection_loss‚2.0842635929584503 .loss/FRt3/detection_loss‚2.580568592995405 /loss/FBt2/detection_loss‚2.2523875615813513 %loss/total_loss‚40.33835871314365 % _timestamp‚1.7884974637829654e+09 2loss/SRqqt2/assignment_loss‚1.4531628263719154 .loss/FRt4/detection_loss‚1.341972102019666 /loss/FBt3/detection_loss‚3.7518510937690737 /loss/FBt4/detection_loss‚0.3775228271439958 _step‚736 0loss/FRt1/assignment_loss‚2.7259705116351447 /loss/FRt4/assignment_loss‚1.243328775184742 2loss/SRqqt4/assignment_loss‚0.7456171344677659 0loss/FBt1/assignment_loss‚0.3568819748208106 1loss/FBt4/assignment_loss‚0.19645873002626757 1loss/SRqqt1/detection_loss‚2.3105824487283826 trainer/global_step‚8949 _runtime‚9563.423380941 0loss/FRt2/assignment_loss‚3.4224554364745683 /loss/FRt3/assignment_loss‚2.723410854116082 2loss/SRqqt1/assignment_loss‚1.0011370098218322àqƒ•Þ’’%‚(:‡ ¾œéÔ³ˆ™° gpu.0.gpu‚38.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚131.496' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.2992 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BaŠ“%‚(:ÿ ¾œéÔÁŽ£± network.sent‚ 2.2642203e+07 network.recv‚ 3.156014e+06) disk./.usagePercent‚78.83868397954824$ disk.dm-0.out‚226871.80126953125 disk.drbd1260.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2949.64453125* proc.memory.percent‚18.003201484680176$ disk.nvme0c0n1.in‚30650.31640625" disk.nvme0n1.in‚30650.31640625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 614.80859375 disk.drbd1260.out‚ 1.3515625 disk.sda.out‚ 614.80859375( proc.memory.availableMB‚ 1119.7890625% disk./.usageGB‚1174.7533645629883) disk.nvme0c0n1.out‚226871.80126953125 disk.sda3.out‚ 614.80859375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.295247703979147' disk.nvme0n1.out‚226871.80126953125 disk.dm-0.in‚30650.31640625$ memory_percent‚93.16534996032715 proc.cpu.threads‚17#4õ”%‚(:’ ÍœéÔ†Šµ¯ gpu.0.gpu‚39.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.918' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.183600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!r~•%‚(:ó ÍœéÔÿÒ‡´ proc.cpu.threads‚17 network.recv‚ 3.162477e+06% disk.nvme0c0n1.out‚227320.6015625 disk.dm-1.in‚0 disk.sda.out‚ 615.29296875 disk.sda1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 1119.5390625 network.sent‚ 2.2751599e+07) disk./.usagePercent‚78.83868397954824% disk./.usageGB‚1174.7533645629883$ disk.nvme0c0n1.in‚30650.31640625# disk.nvme0n1.out‚227320.6015625 disk.dm-0.in‚30650.31640625 disk.sda3.out‚ 615.29296875# memory_percent‚93.1668758392334" disk.nvme0n1.in‚30650.31640625 disk.dm-1.out‚ 615.29296875 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2949.64453125* proc.memory.percent‚18.003201484680176 cpu‚25.294929764092664 disk.dm-0.out‚227320.6015625 disk.sda3.in‚0 disk.drbd1260.out‚ 1.35546875 disk.sda.in‚0¦í–%‚(:’ ÜœéÔŠ­ˆ® gpu.0.gpu‚36.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.791000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QeÌšˆ—%‚(:ý ÜœéԨ솱( disk./.usagePercent‚78.8723697069111 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 network.sent‚ 2.2759331e+07 disk.drbd1260.out‚ 1.35546875 disk.sda.out‚ 615.9296875" disk.nvme0n1.in‚30650.31640625' disk.nvme0n1.out‚227769.39990234375 disk.dm-0.in‚30650.31640625$ disk.dm-0.out‚227769.39990234375 disk.dm-1.in‚0 network.recv‚ 3.166875e+06% disk./.usageGB‚1175.2553062438965 disk.sda1.out‚0 disk.sda3.in‚0$ memory_percent‚93.16685199737549# proc.memory.rssMB‚ 2949.64453125* proc.memory.percent‚18.003201484680176 proc.cpu.threads‚17 disk.sda3.out‚ 615.9296875 disk.dm-1.out‚ 615.9296875) proc.memory.availableMB‚ 1119.54296875 cpu‚25.294669651127084 disk.sda2.out‚0$ disk.nvme0c0n1.in‚30650.31640625) disk.nvme0c0n1.out‚227769.39990234375:aéK2˜%- _wandbruntime‚9613  _runtime‚9613’úIÇ™%Á % _timestamp‚1.7884975138995786e+09 _runtime‚9613.539964106 _step‚737 & lr-AdamW/pg1‚0.0001475501665362887 & lr-AdamW/pg2‚0.0001475501665362887 trainer/global_step‚8999*%MÌš%Æ % _timestamp‚1.7884975138995786e+09 _runtime‚9613.539964106 _step‚737 & lr-AdamW/pg1‚0.0001475501665362887 & lr-AdamW/pg2‚0.0001475501665362887 trainer/global_step‚8999áNp2›%- _wandbruntime‚9613  _runtime‚9613Fúb!û'œ%õO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _step‚738 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9250885669888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚629371963271680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140462817 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90357595 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚698870955780096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9217582958624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 trainer/global_step‚8999 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚638622866944512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9645762000384 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199665974 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚613268929606128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚638622076296704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9645555275264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _runtime‚ 9613.54112801 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140463368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90357597 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199665974 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157595724 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50105771 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚613268912566768 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚629371981274624 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9250885669888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157595724 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚698870931840000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206725120 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357284714 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚638622076296704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚629371981274624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199688990 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230665216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9250113025024 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9250113025024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312tÙ,(%úO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚629371981274624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199688990 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230665216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9250113025024 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9250113025024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _step‚738 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9250885669888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚629371963271680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140462817 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90357595 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚698870955780096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9217582958624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 trainer/global_step‚8999 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚638622866944512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9645762000384 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199665974 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚613268929606128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚638622076296704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9645555275264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _runtime‚ 9613.54112801 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140463368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90357597 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199665974 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157595724 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50105771 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚613268912566768 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚629371981274624 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9250885669888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157595724 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚698870931840000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206725120 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357284714 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚638622076296704ââ¼û¡2ž%- _wandbruntime‚9614  _runtime‚9614îl¯èŸ%€N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199687685 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9217813450090 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9251117770752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140478328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90367243 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50111083 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚622553580140598 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157612598 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157612596 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9251890428928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140477783 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚708593100571136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 % _timestamp‚1.7884975148661015e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9251117770752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚629439681213952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 _runtime‚9614.507775333 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357323325 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚638691571642880 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚708592874113536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚698946280182784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚638690780981760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 C=DeviceStatsMoâu†½nitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9218584098710 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157612596 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚698946304122880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 trainer/global_step‚8999 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚629439663211008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚638691571642880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚629439663211008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚613334996041888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚613334979002528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9251890428928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199710727 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9646796448256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 199710727 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357323325 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199687685 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157612598 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚629439663211008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚638691571642880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚629439663211008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚613334996041888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚613334979002528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9251890428928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199710727 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9646796448256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 199710727 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357323325 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199687685 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157612598 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157612598 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157612596 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9251890428928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140477783 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚708593100571136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 % _timestamp‚1.7884975148661015e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9251117770752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚629439681213952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 _runtime‚9614.507775333 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357323325 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚638691571642880 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚708592874113536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825ãÁrÇh2¡%- _wandbruntime‚9614  _runtime‚9614&·ÎY¢%Ó _step‚740 0loss/SRqqt1/assignment_loss‚1.27835678848727 /loss/FBt4/detection_loss‚0.4829640001555284 _runtime‚9614.509037402 /loss/FRt2/assignment_loss‚3.253165949332087 /loss/FBt3/detection_loss‚3.2951344922184944 %loss/total_loss‚39.13962229143851 epoch‚7 /loss/FRt3/assignment_loss‚2.421303064658724 2loss/SRqqt2/assignment_loss‚1.6686410182328135 2loss/SRqqt3/assignment_loss‚1.1839684165234483 2loss/SRqqt4/assignment_loss‚0.9794807332732222 1loss/FBt3/assignment_loss‚0.05999441677840878 .loss/FRt2/detection_loss‚2.533544759980618 /loss/FBt2/detection_loss‚2.3158348128199577 1loss/SRqqt2/detection_loss‚2.3472447594006858 0loss/FBt1/assignment_loss‚0.3380836568896108 /loss/FRt3/detection_loss‚2.7574515430064035 1loss/SRqqt4/detection_loss‚0.7883924540205329 /loss/FBt1/detection_loss‚1.8342984387507806 trainer/global_step‚8999 % _timestamp‚1.7884975148687494e+09 0loss/FRt1/assignment_loss‚3.2328651984532675 1loss/FBt2/assignment_loss‚0.20260212149952395 1loss/FBt4/assignment_loss‚0.12803610882104596 -loss/FRt4/detection_loss‚1.23202166061348 0loss/FRt4/assignment_loss‚0.8460167370625397 .loss/FRt1/detection_loss‚2.088290017677678 1loss/SRqqt1/detection_loss‚2.0966026762436174 0loss/SRqqt3/detection_loss‚1.775328466538773 ©…^£%Ø 0loss/SRqqt1/assignment_loss‚1.27835678848727 /loss/FBt4/detection_loss‚0.4829640001555284 _runtime‚9614.509037402 /loss/FRt2/assignment_loss‚3.253165949332087 /loss/FBt3/detection_loss‚3.2951344922184944 %loss/total_loss‚39.13962229143851 epoch‚7 /loss/FRt3/assignment_loss‚2.421303064658724 2loss/SRqqt2/assignment_loss‚1.6686410182328135 2loss/SRqqt3/assignment_loss‚1.1839684165234483 2loss/SRqqt4/assignment_loss‚0.9794807332732222 1loss/FBt3/assignment_loss‚0.05999441677840878 .loss/FRt2/detection_loss‚2.533544759980618 /loss/FBt2/detection_loss‚2.3158348128199577 1loss/SRqqt2/detection_loss‚2.3472447594006858 0loss/FBt1/assignment_loss‚0.3380836568896108 /loss/FRt3/detection_loss‚2.7574515430064035 1loss/SRqqt4/detection_loss‚0.7883924540205329 /loss/FBt1/detection_loss‚1.8342984387507806 trainer/global_step‚8999 % _timestamp‚1.7884975148687494e+09 0loss/FRt1/assignment_loss‚3.2328651984532675 1loss/FBt2/assignment_loss‚0.20260212149952395 1loss/FBt4/assignment_loss‚0.12803610882104596 -loss/FRt4/detection_loss‚1.23202166061348 0loss/FRt4/assignment_loss‚0.8460167370625397 .loss/FRt1/detection_loss‚2.088290017677678 1loss/SRqqt1/detection_loss‚2.0966026762436174 0loss/SRqqt3/detection_loss‚1.775328466538773 _step‚740ä§{›©’¤%‚(:‡ ëœéÔÄüø¯ gpu.0.gpu‚41.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.9994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wp¦ß|¥%‚(:ñ ëœéÔÌç­± cpu‚25.294392660386116) disk./.usagePercent‚78.87223427892549 disk.sda3.out‚ 616.3203125 disk.dm-1.in‚0 disk.drbd1260.out‚1.359375 disk.sda.out‚ 616.3203125$ disk.nvme0c0n1.in‚30650.31640625# memory_percent‚93.1682825088501* proc.memory.percent‚18.003201484680176 network.sent‚ 2.2775558e+07 network.recv‚ 3.170504e+06 disk.sda.in‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚228218.57421875$ disk.nvme0n1.out‚228218.57421875$ disk./.usageGB‚1175.253288269043 disk.sda3.in‚0 disk.drbd1260.in‚0! disk.dm-0.out‚228218.57421875) proc.memory.availableMB‚ 1119.30859375 proc.cpu.threads‚17 disk.dm-1.out‚ 616.3203125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚30650.31640625 disk.dm-0.in‚30650.31640625# proc.memory.rssMB‚ 2949.64453125·t`ƒ¦%‚(:… úœéÔ²ÙÉ­ gpu.0.gpu‚48.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0… é §%‚(:ô úœéÔ´Á¸¶ network.recv‚ 3.175689e+06# disk.nvme0c0n1.in‚ 30826.3828125 disk.dm-0.in‚ 30826.3828125" disk.dm-0.out‚228475.208984375 disk.dm-1.out‚ 616.75390625 disk.sda1.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 1119.2578125 network.sent‚ 2.2870517e+07% disk./.usageGB‚1175.2532997131348! disk.nvme0n1.in‚ 30826.3828125 disk.sda3.in‚0# proc.memory.rssMB‚ 2949.64453125 cpu‚25.29409187928885 proc.cpu.threads‚17' disk.nvme0c0n1.out‚228475.208984375 disk.dm-1.in‚0 disk.drbd1260.out‚1.359375 disk.sda1.in‚0* proc.memory.percent‚18.003201484680176) disk./.usagePercent‚78.87223504694809% disk.nvme0n1.out‚228475.208984375 disk.sda3.out‚ 616.75390625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 616.75390625 disk.sda2.out‚0$ memory_percent‚93.16859245300293ûDž5’¨%‚(:‡ ‰éÔÖŒ£­ gpu.0.gpu‚29.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚174.066' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>òÐã…©%‚(:ú ‰éÔ‹ˆ¶°# disk.dm-0.out‚228852.3310546875 disk.drbd1260.out‚ 1.36328125( proc.memory.availableMB‚ 1119.5078125# proc.memory.rssMB‚ 2949.64453125* proc.memory.percent‚18.003201484680176 disk.sda3.out‚ 617.23828125 disk.dm-1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.2876775e+07) disk./.usagePercent‚78.87223504694809 disk.drbd1260.in‚0 disk.sda.out‚ 617.23828125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 30826.3828125 network.recv‚ 3.180625e+06 disk.dm-0.in‚ 30826.3828125 disk.sda3.in‚0 disk.dm-1.out‚ 617.23828125 disk.sda.in‚0( disk.nvme0c0n1.out‚228852.3310546875$ memory_percent‚93.16706657409668 cpu‚25.293798064570222% disk./.usageGB‚1175.2532997131348! disk.nvme0n1.in‚ 30826.3828125& disk.nvme0n1.out‚228852.33105468755Í»ª%‚(:’ ˜éÔ‡øð® gpu.0.gpu‚22.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.216' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.643199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=1x•…«%‚(:ú ˜éԫ㌰& disk.nvme0n1.out‚229335.9052734375# disk.dm-0.out‚229335.9052734375* proc.memory.percent‚18.003201484680176 cpu‚25.29352701106106 disk.sda.out‚ 617.73046875 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 30826.3828125 disk.sda3.in‚0 disk.drbd1260.out‚ 1.36328125# proc.memory.rssMB‚ 2949.64453125 proc.cpu.threads‚17 network.recv‚ 3.185057e+06 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 30826.3828125 disk.sda3.out‚ 617.73046875 disk.dm-1.in‚0 disk.dm-1.out‚ 617.73046875) proc.memory.availableMB‚ 1119.50390625 network.sent‚ 2.2882778e+07( disk.nvme0c0n1.out‚229335.9052734375! disk.nvme0n1.in‚ 30826.3828125 disk.drbd1260.in‚0$ memory_percent‚93.16709041595459) disk./.usagePercent‚78.87066162464265% disk./.usageGB‚1175.2298545837402 disk.sda1.out‚0Ü–-02¬%- _wandbruntime‚9664  _runtime‚9664kq$È­% $ _timestamp‚1.788497564911552e+09 _runtime‚9664.552032281 _step‚741 ' lr-AdamW/pg1‚0.00014312379270623203 ' lr-AdamW/pg2‚0.00014312379270623203 trainer/global_step‚9049¢úŸbÍ®%Ç ' lr-AdamW/pg1‚0.00014312379270623203 ' lr-AdamW/pg2‚0.00014312379270623203 trainer/global_step‚9049 $ _timestamp‚1.788497564911552e+09 _runtime‚9664.552032281 _step‚741å¨á´z2¯%- _wandbruntime‚9664  _runtime‚9664œÔ•¾!"°%ôO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200754876 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141211568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 trainer/global_step‚9049 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚702638348979200 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 # _timestamp‚1.78849756491278e+09 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90839995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158439422 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50371571 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158439424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158439424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚632756978241024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9301137370112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9301137370112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200754876 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _step‚742 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚642057324963328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 359217316 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚712336333501952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200777892 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚712336136391168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9697960582656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141211035 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚642057324963328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 173170688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚625839117250088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚642058115611136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197110784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50371040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _runtime‚9664.553331961 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90839997 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158439422 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚632756960238080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702638372919296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200777892 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9267653543732 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9300364725248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.curren![…Ùt‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚616572234354768 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9697787411968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚642058115611136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 359194298 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚632756960238080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9300364725248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9267653543732 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9300364725248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚616572234354768 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9697787411968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚642058115611136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 359194298 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚632756960238080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9300364725248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90839995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158439422 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50371571 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158439424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158439424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚632756978241024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9301137370112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9301137370112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200754876 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _step‚742 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚642057324963328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 359217316 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚712336333501952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200777892 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚712336136391168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9697960582656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141211035 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚642057324963328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 173170688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚625839117250088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚642058115611136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 197110784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50371040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _runtime‚9664.553331961 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90839997 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096æ‡Â'Ž2²%- _wandbruntime‚9665  _runtime‚9665ÔÌ#='³%ÿM F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90849643 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9268654683818 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702713697321984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200776587 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚642126029648384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚625906184825726 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9267884035198 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚642126820309504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90849645 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9302142129152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 trainer/global_step‚9049 _step‚743 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 359232883 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚712412514713088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9698817391104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200799629 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9302142129152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 359255927 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9699019908608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 359255927 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚616638317829888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141226604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚712412741170688 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702713721262080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚616638300790528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50376959 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141226065 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0±©+… '´%„N YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9302142129152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 trainer/global_step‚9049 _step‚743 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 359232883 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚712412514713088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9698817391104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200799629 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9302142129152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 359255927 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9699019908608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 359255927 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚616638317829888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141226604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚712412741170688 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702713721262080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚616638300790528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50376959 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141226065 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90849643 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9268654683818 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702713697321984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200776587 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚642126029648384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚625906184825726 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9267884035198 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚642126820309504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90849645 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044ç|M”2µ%- _wandbruntime‚9665  _runtime‚9665Ës„¶%Ï % _timestamp‚1.7884975657980545e+09 2loss/SRqqt1/assignment_loss‚1.6581773702587401 2loss/SRqqt3/assignment_loss‚1.4074553931555287 .loss/FRt3/detection_loss‚2.610293344978024 0loss/SRqqt1/detection_loss‚2.448277621609824 .loss/FBt3/detection_loss‚2.389416698898588 .loss/FRt2/detection_loss‚2.851079812068074 /loss/FBt1/detection_loss‚2.4201956510543825 0loss/FRt4/assignment_loss‚1.1552696814139685 0loss/SRqqt4/detection_loss‚0.768843833317465 %loss/total_loss‚41.59860256387617 0loss/FRt3/assignment_loss‚3.1867380615542915 0loss/SRqqt3/detection_loss‚2.040956070815975 epoch‚7 _step‚744 0loss/FBt2/assignment_loss‚0.0687075754459069 /loss/FRt1/detection_loss‚2.3404754447750746 trainer/global_step‚9049 _runtime‚ 9665.43824968 /loss/FRt1/assignment_loss‚3.482276587188244 2loss/SRqqt2/assignment_loss‚1ØiñÑ.4848162308936157 1loss/FBt1/assignment_loss‚0.20026448827686066 0loss/FBt3/assignment_loss‚0.2731086859971172 0loss/SRqqt2/detection_loss‚2.455127429577612 .loss/FBt2/detection_loss‚2.589755288192204 /loss/FBt4/detection_loss‚0.4840017422783847 /loss/FRt2/assignment_loss‚2.701467929141862 2loss/SRqqt4/assignment_loss‚0.8633079036495629 0loss/FBt4/assignment_loss‚0.1676234556172006 /loss/FRt4/detection_loss‚1.5509662637176613"‚Z·%Ô epoch‚7 _step‚744 0loss/FBt2/assignment_loss‚0.0687075754459069 /loss/FRt1/detection_loss‚2.3404754447750746 trainer/global_step‚9049 _runtime‚ 9665.43824968 /loss/FRt1/assignment_loss‚3.482276587188244 2loss/SRqqt2/assignment_loss‚1.4848162308936157 1loss/FBt1/assignment_loss‚0.20026448827686066 0loss/FBt3/assignment_loss‚0.2731086859971172 0loss/SRqqt2/detection_loss‚2.455127429577612 .loss/FBt2/detection_loss‚2.589755288192204 /loss/FBt4/detection_loss‚0.4840017422783847 /loss/FRt2/assignment_loss‚2.701467929141862 2loss/SRqqt4/assignment_loss‚0.8633079036495629 0loss/FBt4/assignment_loss‚0.1676234556172006 /loss/FRt4/detection_loss‚1.5509662637176613 % _timestamp‚1.7884975657980545e+09 2loss/SRqqt1/assignment_loss‚1.6581773702587401 2loss/SRqqt3/assignment_loss‚1.4074553931555287 .loss/FRt3/detection_loss‚2.610293344978024 0loss/SRqqt1/detection_loss‚2.448277621609824 .loss/FBt3/detection_loss‚2.389416698898588 .loss/FRt2/detection_loss‚2.851079812068074 /loss/FBt1/detection_loss‚2.4201956510543825 0loss/FRt4/assignment_loss‚1.1552696814139685 0loss/SRqqt4/detection_loss‚0.768843833317465 %loss/total_loss‚41.59860256387617 0loss/FRt3/assignment_loss‚3.1867380615542915 0loss/SRqqt3/detection_loss‚2.040956070815975èô¹ˆž¸%‚(:’ §éÔ¯ëî® gpu.0.gpu‚26.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.876' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.975200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë%Å„¹%‚(:ù §éÔÂÙå° disk.sda.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2949.64453125 cpu‚25.293225621857154 network.sent‚ 2.2887822e+07% disk./.usageGB‚1175.2533226013184 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 3.188554e+06 disk.sda3.in‚0 disk.sda3.out‚ 618.3671875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.3671875# disk.nvme0c0n1.in‚ 30826.3828125! disk.nvme0n1.in‚ 30826.3828125' disk.nvme0n1.out‚229628.37353515625) disk./.usagePercent‚78.87223658299331 disk.dm-1.in‚0 disk.dm-1.out‚ 618.3671875 disk.sda1.out‚0) disk.nvme0c0n1.out‚229628.37353515625$ memory_percent‚93.16744804382324( proc.memory.availableMB‚ 1119.4453125* proc.memory.percent‚18.003201484680176 disk.sda.out‚ 618.3671875 disk.dm-0.in‚ 30826.3828125$ disk.dm-0.out‚229628.37353515625Js]„’º%‚(:‡ ¶éÔÀß¼­ gpu.0.gpu‚44.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zTJ7ˆ»%‚(:ý ¶éÔÎÈŽ°$ disk.dm-0.out‚230076.88916015625 disk.sda3.in‚0( proc.memory.availableMB‚ 1119.4453125) disk./.usagePercent‚78.87223530295563) disk.nvme0c0n1.out‚230076.88916015625 disk.dm-0.in‚30972.86328125 disk.dm-1.in‚0 disk.dm-1.out‚ 618.74609375$ memory_percent‚93.16744804382324 network.recv‚ 3.193784e+06" disk.nvme0n1.in‚30972.86328125 disk.drbd1260.in‚0* proc.memory.percent‚18.003201484680176 cpu‚25.29295820246124 proc.cpu.threads‚17 network.sent‚ 2.2995868e+07 disk.sda.in‚0 disk.sda.out‚ 618.74609375 disk.sda2.in‚0' disk.nvme0n1.out‚230076.88916015625 disk.sda3.out‚ 618.74609375 disk.drbd1260.out‚ 1.3671875# proc.memory.rssMB‚ 2949.64453125$ disk./.usageGB‚1175.253303527832 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚30972.86328125«ë’¼%‚(:‡ ÅéÔ¦»Á® gpu.0.gpu‚46.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.831' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0| ”Š½%‚(:ÿ ÅéÔ±×Ö² disk.dm-1.in‚0 disk.drbd1260.in‚0$ disk.dm-0.out‚230501.60107421875( proc.memory.availableMB‚ 1119.4453125 cpu‚25.292674358033896 proc.cpu.threads‚17 network.sent‚ 2.3002187e+07 network.recv‚ 3.198651e+06) disk./.usagePercent‚78.87223530295563 disk.dm-1.out‚ 619.26953125 disk.sda.out‚ 619.26953125 disk.sda2.in‚0) disk.nvme0c0n1.out‚230501.60107421875' disk.nvme0n1.out‚230501.60107421875$ disk./.usageGB‚1175.253303527832 disk.sda3.out‚ 619.26953125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚30972.86328125" disk.nvme0n1.in‚30972.86328125 disk.dm-0.in‚30972.86328125 disk.sda3.in‚0 disk.drbd1260.out‚ 1.37109375 disk.sda2.out‚0$ memory_percent‚93.16744804382324# proc.memory.rssMB‚ 2949.64453125* proc.memory.percent‚18.003201484680176® ]2¾%- _wandbruntime‚9714  _runtime‚9714¡ŽÛçÈ¿% trainer/global_step‚9099 $ _timestamp‚1.788497614926185e+09 _runtime‚9714.566871162 _step‚745 ' lr-AdamW/pg1‚0.00013875014312625392 ' lr-AdamW/pg2‚0.00013875014312625392º0,øÍÀ%Ç _runtime‚9714.566871162 _step‚745 ' lr-AdamW/pg1‚0.00013875014312625392 ' lr-AdamW/pg2‚0.00013875014312625392 trainer/global_step‚9099 $ _timestamp‚1.788497614926185e+09麹O›2Á%- _wandbruntime‚9714  _runtime‚9714'q½ú'Â%ôO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚629192509623196 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50637025 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201843778 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚645492573629952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 361126900 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91322395 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚706405790058496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚629193297310968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9351389070336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚636141957204480 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚619875573182128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9350616425472 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201843778 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201866794 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206725120 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚636141957204480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9317724128840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9316953480428 _runtime‚9714.568343386 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 361149918 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159283122 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚645492573629952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚636141975207424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201866794 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9350616425472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚716155967292416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚645493364277760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9351389070336xX?°ÿ'Ã%ùO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91322397 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9750177233920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚716155736627200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9749970508800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚619875573182128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9350616425472 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201843778 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201866794 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206725120 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚636141957204480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9317724128840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9316953480428 _runtime‚9714.568343386 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 361149918 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159283122 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚645492573629952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚636141975207424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201866794 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9350616425472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚716155967292416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚645493364277760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9351389070336 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚629192509623196 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50637025 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201843778 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚645492573629952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 361126900 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91322395 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚706405790058496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚629193297310968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9351389070336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚636141957204480 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 361188529 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚636209675146752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91332043 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619941622578528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159299996 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159299996 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141974896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9352393829376 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 _step‚747 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9317954620306 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50642322 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9751212806144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚636209675146752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚716232351207424 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚9715.588925739 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚645562068976128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚706481138401280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚629260364886814 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788497615947151e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚636209657143808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 201865489 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9751014482944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚9099 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50642851 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚716232128944128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚645562068976128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚636209657143808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9352393829376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 361188529 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 361165485 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141974365 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141974365 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 361188529 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚636209675146752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91332043 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619941622578528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159299996 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159299996 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141974896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9352393829376 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 _step‚747 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9317954620306 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50642322 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9751212806144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚636209675146752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚716232351207424 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚9715.588925739 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚645562068976128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚706481138401280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚629260364886814 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788497615947151e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚636209657143808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 201865489 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9751014482944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚9099 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50642851 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚716232128944128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005ë‚\2Ç%- _wandbruntime‚9715  _runtime‚9715 ÇMZÈ%Ô /loss/FRt4/detection_loss‚1.4245541542481917 .loss/FBt1/detection_loss‚2.216606989502907 %loss/total_loss‚37.35097461745282 .loss/FRt2/assignment_loss‚2.93732511666086 /loss/FRt3/detection_loss‚2.2351813688874245 0loss/FRt3/assignment_loss‚1.3837416403823428 1loss/SRqqt2/assignment_loss‚1.410584321555992 % _timestamp‚1.7884976159499493e+09 _runtime‚9715.590286123 _step‚748 2loss/SRqqt4/assignment_loss‚0.8673053081812603 .loss/FRt2/detection_loss‚2.341201812148332 0loss/SRqqt2/detection_loss‚2.466318671902021 1loss/SRqqt3/detection_loss‚1.8276264794863935 trainer/global_step‚9099 /loss/FRt1/assignment_loss‚2.577812745127567 2loss/SRqqt3/assignment_loss‚1.1267980531637751 /loss/FBt3/detection_loss‚2.1828530601092746 epoch‚7 1loss/FBt2/assignment_loss‚0.24665689078847208 .loss/FRt1/detection_loss‚2.470607012163761 1loss/FBt3/assignment_loss‚0.20631111159696047 0loss/SRqqt1/detection_loss‚2.158100108037124 1loss/SRqqt4/detection_loss‚0.7588908629758018 /loss/FBt2/detection_loss‚3.2266745269298553 /loss/FBt4/detection_loss‚0.5250881072785706 0loss/FRt4/assignment_loss‚1.1719957635952876 2loss/SRqqt1/assignment_loss‚1.2469670903441068 1loss/FBt1/assignment_loss‚0.17698935233033572 1loss/FBt4/assignment_loss‚0.16478407005620435ÝðÕ_É%Ù /loss/FRt3/detection_loss‚2.2351813688874245 0loss/FRt3/assignment_loss‚1.3837416403823428 1loss/SRqqt2/assignment_loss‚1.410584321555992 % _timestamp‚1.7884976159499493e+09 _runtime‚9715.590286123 _step‚748 2loss/SRqqt4/assignment_loss‚0.8673053081812603 .loss/FRt2/detection_loss‚2.341201812148332 0loss/SRqqt2/detection_loss‚2.466318671902021 1loss/SRqqt3/detection_loss‚1.8276264794863935 trainer/global_step‚9099 /loss/FRt1/assignment_loss‚2.577812745127567 2loss/SRqqt3/assignment_loss‚1.1267980531637751 /loss/FBt3/detection_loss‚2.1828530601092746 epoch‚7 1loss/FBt2/assignment_loss‚0.24665689078847208 .loss/FRt1/detection_loss‚2.470607012163761 1loss/FBt3/assignment_loss‚0.20631111159696047 0loss/SRqqt1/detection_loss‚2.158100108037124 1loss/SRqqt4/detection_loss‚0.7588908629758018 /loss/FBt2/detection_loss‚3.2266745269298553 /loss/FBt4/detection_loss‚0.5250881072785706 0loss/FRt4/assignment_loss‚1.1719957635952876 2loss/SRqqt1/assignment_loss‚1.2469670903441068 1loss/FBt1/assignment_loss‚0.17698935233033572 1loss/FBt4/assignment_loss‚0.16478407005620435 /loss/FRt4/detection_loss‚1.4245541542481917 .loss/FBt1/detection_loss‚2.216606989502907 %loss/total_loss‚37.35097461745282 .loss/FRt2/assignment_loss‚2.93732511666086ìÇ?ª’Ê%‚(:‡ ÔéÔ·°¦¯ gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñè‡Ë%‚(:ü ÔéÔÎ̲ network.sent‚ 2.3008155e+07) disk./.usagePercent‚78.87223530295563 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-0.in‚30972.86328125* proc.memory.percent‚18.003201484680176 cpu‚25.292430884694678 proc.cpu.threads‚17$ disk./.usageGB‚1175.253303527832 disk.dm-1.out‚ 620.00390625 disk.drbd1260.in‚0 disk.sda.out‚ 620.00390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 1.37109375$ disk.nvme0c0n1.in‚30972.86328125" disk.nvme0n1.in‚30972.86328125' disk.nvme0n1.out‚230926.20654296875$ disk.dm-0.out‚230926.20654296875& proc.memory.availableMB‚ 1119.15625# proc.memory.rssMB‚ 2949.64453125 network.recv‚ 3.203381e+06 disk.sda3.out‚ 620.00390625 disk.dm-1.in‚0) disk.nvme0c0n1.out‚230926.20654296875# memory_percent‚93.1692123413086Ø"¦6’Ì%‚(:‡ ãéÔ¿Íé® gpu.0.gpu‚31.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.194' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6388 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í'u%‚Í%‚(:÷ ãéÔ¢¿Ù³' disk.nvme0n1.out‚231253.69287109375( proc.memory.availableMB‚ 1119.3984375# proc.memory.rssMB‚ 2949.64453125 cpu‚25.292131045646492$ disk.nvme0c0n1.in‚30972.86328125 disk.dm-0.in‚30972.86328125$ disk.dm-0.out‚231253.69287109375$ memory_percent‚93.16773414611816* proc.memory.percent‚18.003201484680176 network.sent‚ 2.3116206e+07 network.recv‚ 3.208796e+06$ disk./.usageGB‚1175.253303527832 disk.sda3.in‚0 disk.sda3.out‚ 620.3203125 disk.dm-1.out‚ 620.3203125 proc.cpu.threads‚17) disk./.usagePercent‚78.87223530295563 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 620.3203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚1.375 disk.sda2.out‚0) disk.nvme0c0n1.out‚231253.69287109375" disk.nvme0n1.in‚30972.86328125‹2º‰Î%‚(:… òéÔ§Ù³° gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F/ºû„Ï%‚(:ù òéÔƒ¦Ä³ disk.sda.out‚ 620.73828125% disk./.usageGB‚1175.2415771484375 disk.dm-0.in‚ 31004.15625$ disk.dm-0.out‚231702.18896484375 disk.dm-1.in‚0 disk.drbd1260.out‚1.375 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 2.3121292e+07 network.recv‚ 3.212335e+06' disk.nvme0n1.out‚231702.18896484375* proc.memory.percent‚18.003201484680176 cpu‚25.291838724982288 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.out‚ 620.73828125 disk.sda.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚31004.53515625) disk.nvme0c0n1.out‚231702.18896484375) disk./.usagePercent‚78.87144833579538 disk.sda3.out‚ 620.73828125 disk.drbd1260.in‚0 disk.sda1.in‚0" disk.nvme0n1.in‚31004.66015625$ memory_percent‚93.16771030426025) proc.memory.availableMB‚ 1119.40234375# proc.memory.rssMB‚ 2949.64453125zDufÐ%‚(:’ žéÔ’·¦¯ gpu.0.gpu‚43.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.759' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.151799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ºö—Ñ%‚(:ö žéÔÝÇ”´ disk.sda.in‚0# proc.memory.rssMB‚ 2949.64453125 cpu‚25.29159436212238 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚18.003201484680176 network.recv‚ 3.217713e+06$ disk./.usageGB‚1175.255027770996) disk.nvme0c0n1.out‚232152.87060546875! disk.nvme0n1.in‚ 31102.8515625$ disk.dm-0.out‚232152.87060546875 disk.sda3.out‚ 621.3359375 disk.drbd1260.out‚ 1.37890625 disk.sda.out‚ 621.3359375 network.sent‚ 2.3127764e+07) disk./.usagePercent‚78.87235101836112' disk.nvme0n1.out‚232152.87060546875 disk.dm-0.in‚ 31102.8515625 disk.dm-1.out‚ 621.3359375 disk.sda1.in‚0$ memory_percent‚93.16768646240234& proc.memory.availableMB‚ 1119.40625 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 31102.8515625 disk.sda3.in‚0ü ¯}2Ò%- _wandbruntime‚9765  _runtime‚9765õõ´ÉÓ%à trainer/global_step‚9149 % _timestamp‚1.7884976656510983e+09 _runtime‚9765.291493085 _step‚749 ' lr-AdamW/pg1‚0.00013443012925837166 ' lr-AdamW/pg2‚0.00013443012925837166ˆ‹Ù¯ÎÔ%È ' lr-AdamW/pg1‚0.00013443012925837166 ' lr-AdamW/pg2‚0.00013443012925837166 trainer/global_step‚9149 % _timestamp‚1.7884976656510983e+09 _runtime‚9765.291493085 _step‚749íÈÅg2Õ%- _wandbruntime‚9765  _runtime‚9765u©c'Ö%öO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202932680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142706083 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9802348826624 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9401640770560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50900748 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚648928612944384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623178894970128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚750 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9400868125696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9367024065536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚632546689684076 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚648927822296576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚639526954170880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚623178877930768 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202955696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202955696 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202932680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚719975532084224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9802545065984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚648928612944384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160126822 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚648927822296576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 363059502 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 160126824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚639526972173824 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚632545901996304 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚639526972173824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 363082520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9367794713948 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _runtime‚9765.293049248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚710173183257600 trainer/global_step‚9149 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9401640770560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160126822 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50901286 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 KBDeviceStatsMonitoáÅ)är.on_train_batch_start/allocation.large_pool.peak‚3125 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489±dMä(×%ûO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚719975532084224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9802545065984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚648928612944384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160126822 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚648927822296576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 363059502 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 160126824 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚639526972173824 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚632545901996304 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚639526972173824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 363082520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9367794713948 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _runtime‚9765.293049248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚710173183257600 trainer/global_step‚9149 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9401640770560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160126822 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50901286 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202932680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142706083 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9802348826624 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9401640770560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50900748 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚648928612944384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623178894970128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚750 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9400868125696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9367024065536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚632546689684076 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚648927822296576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚639526954170880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚623178877930768 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202955696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202955696 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202932680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328îõ )/2Ø%- _wandbruntime‚9766  _runtime‚9766§”>M'Ù%€N ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9803377385472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9401872871424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 % _timestamp‚1.7884976666379142e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 363098087 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 trainer/global_step‚9149 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9368025205414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 363121131 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202954391 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206711808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202977433 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚639594654110208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160143698 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚710248531600384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚632613757259922 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 363098087 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9368795854034 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50905960 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 363121131 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142720403 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91814445 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9803584097280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚648997317642752 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚648996526981632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9402645529600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚720052139637760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚639594672113152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230651904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚551 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91814443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50906511tcv 'Ú%…N E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚648997317642752 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚648996526981632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9402645529600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚720052139637760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚639594672113152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230651904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚551 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91814443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50906511 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9803377385472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9401872871424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 % _timestamp‚1.7884976666379142e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 363098087 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 trainer/global_step‚9149 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9368025205414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 363121131 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202954391 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206711808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202977433 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚639594654110208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160143698 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚710248531600384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚632613757259922 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 363098087 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9368795854034 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50905960 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 363121131 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142720403 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91814445 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9803584097280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992ïW°e2Û%- _wandbruntime‚9766  _runtime‚9766{›ç$\Ü%Ö 0loss/FRt3/assignment_loss‚2.6838987708091735 2loss/SRqqt1/assignment_loss‚1.4226608059058587 2loss/SRqqt4/assignment_loss‚0.8821699716776076 /loss/FRt2/detection_loss‚2.6272399254932233 _runtime‚9766.280540645 2loss/FBt3/assignment_loss‚0.019603590680911203 1loss/FBt4/assignment_loss‚0.11380937518859385 .loss/FRt1/detection_loss‚2.286404371727258 .loss/FRt3/detection_loss‚2.721121238172054 1loss/SRqqt4/detection_loss‚0.8488609355990775 $ _timestamp‚1.788497666640243e+09 1loss/SRqqt2/detection_loss‚2.4861123846335844 _step‚752 2loss/SRqqt3/assignment_loss‚1.1739646751611006 /loss/FBt4/detection_loss‚0.6282973555704195 0loss/FRt4/assignment_loss‚1.2687496145566304 0loss/SRqqt1/detection_loss‚2.053140769402186 /loss/FBt3/detection_loss‚1.7970241010189056 .loss/FBt1/detection_loss‚2.527412865843092 trainer/global_step‚9149 2loss/SRqqt2/assignment_loss‚1.6330170170827345 0loss/FBt1/assignment_loss‚0.1126367487678073 1loss/FBt2/assignment_loss‚0.10832981085372416 .loss/FRt4/detection_loss‚1.371764146231322 1loss/SRqqt3/detection_loss‚1.8501827260035844 %loss/total_loss‚39.06613211833633 epoch‚7 /loss/FRt1/assignment_loss‚2.877550847828388 0loss/FRt2/assignment_loss‚3.1562927540610817 /loss/FBt2/detection_loss‚2.4158873160680137Œj åÝ%Û 1loss/SRqqt2/detection_loss‚2.4861123846335844 _step‚752 2loss/SRqqt3/assignment_loss‚1.1739646751611006 /loss/FBt4/detection_loss‚0.6282973555704195 0loss/FRt4/assignment_loss‚1.2687496145566304 0loss/SRqqt1/detection_loss‚2.053140769402186 /loss/FBt3/detection_loss‚1.7970241010189056 .loss/FBt1/detection_loss‚2.527412865843092 trainer/global_step‚9149 2loss/SRqqt2/assignment_loss‚1.6330170170827345 0loss/FBt1/assignment_loss‚0.1126367487678073 1loss/FBt2/assignment_loss‚0.10832981085372416 .loss/FRt4/detection_loss‚1.371764146231322 1loss/SRqqt3/detection_loss‚1.8501827260035844 %loss/total_loss‚39.06613211833633 epoch‚7 /loss/FRt1/assignment_loss‚2.877550847828388 0loss/FRt2/assignment_loss‚3øðœO.1562927540610817 /loss/FBt2/detection_loss‚2.4158873160680137 0loss/FRt3/assignment_loss‚2.6838987708091735 2loss/SRqqt1/assignment_loss‚1.4226608059058587 2loss/SRqqt4/assignment_loss‚0.8821699716776076 /loss/FRt2/detection_loss‚2.6272399254932233 _runtime‚9766.280540645 2loss/FBt3/assignment_loss‚0.019603590680911203 1loss/FBt4/assignment_loss‚0.11380937518859385 .loss/FRt1/detection_loss‚2.286404371727258 .loss/FRt3/detection_loss‚2.721121238172054 1loss/SRqqt4/detection_loss‚0.8488609355990775 $ _timestamp‚1.788497666640243e+09ðééÞ%‚(:’ žéÔí‹Ë° gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.939' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.587799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HŠ¥‚ß%‚(:÷ žéÔçÞü² disk.dm-0.in‚ 31102.8515625 disk.dm-1.out‚ 621.8203125& proc.memory.availableMB‚ 1119.09375# proc.memory.rssMB‚ 2949.64453125 proc.cpu.threads‚17) disk.nvme0c0n1.out‚232345.44482421875 disk.sda3.out‚ 621.8203125 disk.dm-1.in‚0 disk.sda.out‚ 621.8203125 disk.sda1.in‚0* proc.memory.percent‚18.003201484680176 cpu‚25.291308657568575 network.recv‚ 3.222144e+06$ disk./.usageGB‚1175.255027770996# disk.nvme0c0n1.in‚ 31102.8515625! disk.nvme0n1.in‚ 31102.8515625' disk.nvme0n1.out‚232345.44482421875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 network.sent‚ 2.3133562e+07) disk./.usagePercent‚78.87235101836112$ disk.dm-0.out‚232345.44482421875 disk.drbd1260.out‚ 1.37890625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚93.16959381103516çìΑà%‚(:† ŸžéÔÞ¡‚¯ gpu.0.gpu‚25.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.899 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°Mz{á%‚(:ð ŸžéÔéÕë²$ disk./.usageGB‚1175.255027770996 disk.sda2.out‚0) disk.nvme0c0n1.out‚232746.35302734375 disk.dm-1.in‚0# proc.memory.rssMB‚ 2949.64453125 cpu‚25.291044151474146 network.recv‚ 3.227485e+06) disk./.usagePercent‚78.87235101836112! disk.nvme0n1.in‚ 31102.8515625' disk.nvme0n1.out‚232746.35302734375 disk.dm-1.out‚622.1875$ memory_percent‚93.16961765289307* proc.memory.percent‚18.003201484680176 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 31102.8515625 disk.sda3.in‚0 disk.sda3.out‚622.1875 proc.cpu.threads‚17 network.sent‚ 2.3241623e+07 disk.sda.out‚622.1875 disk.sda1.in‚0 disk.dm-0.in‚ 31102.8515625$ disk.dm-0.out‚232746.35302734375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.3828125) proc.memory.availableMB‚ 1119.08984375Šÿ Öâ%‚(:’ ®žéÔæ肯 gpu.0.gpu‚20.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.119799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÆ ‚ã%‚(:÷ ®žéÔόس disk.sda3.out‚ 622.8359375 disk.dm-1.in‚0 disk.dm-1.out‚ 622.8359375$ memory_percent‚93.16811561584473( proc.memory.availableMB‚ 1119.3359375 network.sent‚ 2.3246773e+07) disk./.usagePercent‚78.87235101836112# disk.nvme0c0n1.in‚ 31102.8515625) disk.nvme0c0n1.out‚233073.68115234375' disk.nvme0n1.out‚233073.68115234375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 network.recv‚ 3.231093e+06$ disk./.usageGB‚1175.255027770996 disk.sda2.out‚0# proc.memory.rssMB‚ 2949.64453125* proc.memory.percent‚18.003201484680176 cpu‚25.29080048114175 proc.cpu.threads‚17! disk.nvme0n1.in‚ 31102.8515625 disk.dm-0.in‚ 31102.8515625 disk.drbd1260.out‚ 1.3828125 disk.sda.out‚ 622.8359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0$ disk.dm-0.out‚233073.68115234375@Öì]2ä%- _wandbruntime‚9816  _runtime‚9816³9 Éå%à ' lr-AdamW/pg1‚0.00013016465138700003 ' lr-AdamW/pg2‚0.00013016465138700003 trainer/global_step‚9199 % _timestamp‚1.7884977164376516e+09 _runtime‚9816.078087681 _step‚753©ÜÎæ%È trainer/global_step‚9199 % _timestamp‚1.7884977164376516e+09 _runtime‚9816.078087681 _step‚753 ' lr-AdamW/pg1‚0.00013016465138700003 ' lr-AdamW/pg2‚0.00013016465138700003ñ4¡u2ç%- _wandbruntime‚9816  _runtime‚9816%oÇü'è%öO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9417865299056 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364992104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚642911951137280 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚652363070963200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635899294369412 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9417094650644 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚642911951137280 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 204021582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51165502 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚626482216758128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚9816.079277994 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 365015122 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51166053 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚635900082057184 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9451892470784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚723795591829504 _step‚754 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9451119825920 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚652363861611008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160970522 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚553 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚642911969140224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9854777544704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 204021582 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204044598 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204044598 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚652363861611008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143452697 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9451892470784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚723795377941504 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚713940600396800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 trainer/global_step‚9199 % _timestamp‚1.7884977164388063e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9451119825920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213888000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚652363070963200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚626482199718768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160970522×ðd(é%ûO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9451892470784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚723795591829504 _step‚754 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9451119825920 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚652363861611008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160970522 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚553 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚642911969140224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9854777544704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 204021582 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204044598 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204044598 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚652363861611008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143452697 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9451892470784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚723795377941504 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚713940600396800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 trainer/global_step‚9199 % _timestamp‚1.7884977164388063e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9451119825920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213888000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚652363070963200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚626482199718768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160970522 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9417865299056 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364992104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚642911951137280 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚652363070963200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635899294369412 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9417094650644 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚642911951137280 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 204021582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51165502 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚626482216758128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚9816.079277994 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 365015122 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51166053 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚635900082057184 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516ò¶U9i2ê%- _wandbruntime‚9816  _runtime‚9816BsÔë%€N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160987396 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143467639 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9418866439142 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51171340 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9855831515136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9816.963251608 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9452897229824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚626548266154528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _step‚755 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204066335 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚642979651076608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51170796 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚652432566309376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 365053733 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 365053733 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160987398 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9856023546880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160987398 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 % _timestamp‚1.7884977173224306e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92296843 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9452124571648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚635966361945050 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204043293 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9452897229824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚652431775648256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 trainer/global_step‚9199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚642979651076608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚642979669079552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 % _timestamp‚1.7884977173224306e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92296843 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9452124571648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚635966361945050 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204043293 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9452897229824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚652431775648256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 trainer/global_step‚9199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚642979651076608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚642979669079552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143467639 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9418866439142 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51171340 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9855831515136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚9816.963251608 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9452897229824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚626548266154528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _step‚755 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204066335 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚642979651076608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51170796 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚652432566309376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 365053733 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 365053733 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160987398 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9856023546880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992óM”Ss2í%- _wandbruntime‚9816  _runtime‚9816^¡ @Yî%Ó /loss/FRt3/assignment_loss‚2.186734262350443 2loss/SRqqt1/assignment_loss‚1.4343609466132792 1loss/SRqqt2/assignment_loss‚1.573040158878411 /loss/FRt3/detection_loss‚2.5447665976511464 epoch‚7 % _timestamp‚1.7884977173246055e+09 _runtime‚ 9816.96479095 /loss/FRt2/assignment_loss‚3.697139330448643 0loss/FBt2/assignment_loss‚0.3425450080443018 .loss/FRt2/detection_loss‚2.245361393223697 .loss/FBt1/detection_loss‚2.337025450808661 /loss/FBt2/detection_loss‚2.5580897861056857 &loss/total_loss‚38.768712060062654 trainer/global_step‚9199 /loss/FRt4/detection_loss‚1.5500959944750727 1loss/SRqqt3/detection_loss‚1.8420632354915143 .loss/FBt3/detection_loss‚2.098590029610528 0loss/FRt1/assignment_loss‚3.2328974434307645 1loss/FBt3/assignment_loss‚0.16403355449674312 0loss/FBt4/assignment_loss‚0.0844500225090952 _step‚756 0loss/FRt4/assignment_loss‚1.0616128105541756 1loss/SRqqt1/detection_loss‚1.9265295924110846 0loss/SRqqt4/detection_loss‚0.920279798263477 1loss/FBt1/assignment_loss‚0.07415110693209925 1loss/SRqqt2/detection_loss‚2.2378801991862636 1loss/SRqqt3/assignment_loss‚1.245014267116785 2loss/SRqqt4/assignment_loss‚0.9636865662359115 0loss/FBt4/detection_loss‚0.43182495275411653 .loss/FRt1/detection_loss‚2.016539552470758ÿË^ï%Ø 0loss/FBt2/assignment_loss‚0.3425450080443018 .loss/FRt2/detection_loss‚2.245361393223697 .loss/FBt1/detection_loss‚2.337025450808661 /loss/FBt2/detection_loss‚2.5580897861056857 &loss/total_loss‚38.768712060062654 trainer/global_step‚9199 /loss/FRt4/detection_loss‚1.5500959944750727 1loss/SRqqt3/detection_loss‚1.8420632354915143 .loss/FBt3/detection_loss‚2.098590029610528 0loss/FRt1/assignment_loss‚3.2328974434307645 1loss/FBt3/assignment_loss‚0.16403355449674312 0loss/FBt4/assignment_loss‚0.0844500225090952 _step‚756 0loss/FRt4/assignment_loss‚1.0616128105541756 1loss/SRqqt1/detection_loss‚1.9265295924110846 0loss/SRqqt4/detection_loss‚0.920279798263477 1loss/FBt1/assignment_loss‚0.07415110693209925 1loss/SRqqt2/detection_loss‚2.2378801991862636 1loss/SRqqt3/assignment_loss‚1.245014267116785 2loss/SRqqt4/assignment_loss‚0.9636865662359115 0loss/FBt4/detection_loss‚0.43182495275411653 .loss/FRt1/detection_loss‚2.016539552470758 /loss/FRt3/assignment_loss‚2.186734262350443 2loss/SRqqt1/assignment_loss‚1.4343609466132792 1loss/SRqqt2/assignment_loss‚1.573040158878411 /loss/FRt3/detection_loss‚2.5447665976511464 epoch‚7 % _timestamp‚1.7884977173246055e+09 _runtime‚ 9816.96479095 /loss/FRt2/assignment_loss‚3.697139330448643ôwrÎOð%‚(:’ ½žéÔý족 gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.596' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.119200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o°ê~ñ%‚(:ó ½žéÔá’ð±# memory_percent‚93.1697130203247 disk.sda3.out‚ 623.359375 disk.drbd1260.out‚ 1.38671875 disk.sda.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚233678.81982421875 disk.dm-0.in‚ 31230.0703125) proc.memory.availableMB‚ 1119.07421875# proc.memory.rssMB‚ 2949.64453125 network.sent‚ 2.325301e+07% disk./.usageGB‚1175.2550506591797 disk.sda.out‚ 623.359375 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 31230.0703125 proc.cpu.threads‚17 network.recv‚ 3.23613e+06( disk./.usagePercent‚78.8723525544063 disk.sda3.in‚0 disk.dm-1.in‚0) disk.nvme0c0n1.out‚233678.81982421875* proc.memory.percent‚18.003201484680176 cpu‚25.29054621841636 disk.dm-1.out‚ 623.359375 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 31230.0703125$ disk.dm-0.out‚233678.81982421875½Êx´ò%‚(:’ ÌžéÔƒñ³ gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚148.975' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.794999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í÷öâ€ó%‚(:õ ÌžéÔƒý±½* proc.memory.percent‚18.003201484680176 proc.cpu.threads‚17 network.recv‚ 3.242505e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 623.78125 disk.drbd1260.in‚0 disk.sda.out‚ 623.78125) disk./.usagePercent‚78.87235537048919 disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 31230.0703125$ disk.dm-0.out‚233971.09716796875# proc.memory.rssMB‚ 2949.64453125 cpu‚25.290247622897436 network.sent‚ 2.3361884e+07% disk./.usageGB‚1175.2550926208496 disk.drbd1260.out‚ 1.38671875 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚233971.09716796875$ memory_percent‚93.16980838775635 disk.sda3.out‚ 623.78125 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 31230.0703125! disk.nvme0n1.in‚ 31230.0703125' disk.nvme0n1.out‚233971.09716796875) proc.memory.availableMB‚ 1119.05859375b€û’ô%‚(:‡ ÛžéÔàÆÛ­ gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.558' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.1116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚‹Ýtõ%‚(:é ÛžéÔîýŽ° disk.dm-0.out‚ 234371.65625 disk.sda3.in‚0 disk.drbd1260.out‚1.390625) proc.memory.availableMB‚ 1119.01171875* proc.memory.percent‚18.003201484680176 cpu‚25.29000530598935 network.recv‚ 3.245981e+06) disk./.usagePercent‚78.87211037127892% disk./.usageGB‚1175.2514419555664 disk.sda.in‚0# disk.nvme0c0n1.in‚ 31230.0703125! disk.nvme0n1.in‚ 31230.0703125 disk.dm-1.out‚ 624.24609375 proc.cpu.threads‚17 network.sent‚ 2.3366997e+07 disk.sda.out‚ 624.24609375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 624.24609375 disk.sda1.out‚0# disk.nvme0c0n1.out‚ 234371.65625 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚93.17009449005127# proc.memory.rssMB‚ 2949.64453125! disk.nvme0n1.out‚ 234371.65625 disk.dm-0.in‚ 31230.0703125IT42ö%- _wandbruntime‚9867  _runtime‚9867t9CÉ÷%à ' lr-AdamW/pg1‚0.00012595459843133436 ' lr-AdamW/pg2‚0.00012595459843133436 trainer/global_step‚9249 % _timestamp‚1.7884977677686293e+09 _runtime‚9867.409062009 _step‚757Âc¾WÎø%È _runtime‚9867.409062009 _step‚757 ' lr-AdamW/pg1‚0.00012595459843133436 ' lr-AdamW/pg2‚0.00012595459843133436 trainer/global_step‚9249 % _timestamp‚1.7884977677686293e+09õðåÙ2ù%- _wandbruntime‚9867  _runtime‚9867“É®øü'ú%öO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161814224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚655798319629824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 205110484 _step‚758 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144202739 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚646296966106624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚655799110277632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 % _timestamp‚1.7884977677697482e+09 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚717708041476096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚655799110277632 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚646296966106624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366947724 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92769595 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9467935884164 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚9867.410278913 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9502144171008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9501371526144 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144202180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚557 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚639252686742520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92769597 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚727615248548864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205133500 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185753600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161814222 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161814224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9907416766464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚629785521506768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 205133500 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9502144171008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚655798319629824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚559’‰˜b. û%ûO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚655799110277632 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚646296966106624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366947724 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92769595 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9467935884164 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚9867.410278913 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9502144171008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9501371526144 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144202180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚557 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚639252686742520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes)íTÓ.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92769597 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚727615248548864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205133500 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185753600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161814222 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161814224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9907416766464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚629785521506768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 205133500 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9502144171008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚655798319629824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚559 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161814224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚655798319629824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 205110484 _step‚758 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144202739 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚646296966106624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚655799110277632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 % _timestamp‚1.7884977677697482e+09 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚717708041476096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624öÜR¼2ü%- _wandbruntime‚9868  _runtime‚9868ÿ å'ý%þM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚717783365878784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9502376271872 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228554752 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9503148930048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9502376271872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚727691636687872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639320542006138 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚646364648043008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92779245 _runtime‚ 9868.2734459 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚646364666045952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚655867814976000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51437911 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9908270809088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚717783389818880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚639319754318158 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9468937024250 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366963291 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚655867024314880 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161831098 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144217154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚646364648043008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚655867814976000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 trainer/global_step‚9249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161831096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚556 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161831096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 366963291 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚646364666045952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366986335 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144217710 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚639319754318158 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9468937024250 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366963291 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚655867024314880 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161831098 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144217154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚646364648043008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚655867814976000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 trainer/global_step‚9249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161831096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚556 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161831096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 366963291 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚646364666045952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366986335 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144217710 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚717783365878784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9502376271872 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228554752 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9503148930048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9502376271872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚727691636687872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639320542006138 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚646364648043008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92779245 _runtime‚ 9868.2734459 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚646364666045952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚655867814976000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0÷¨fËö2ÿ%- _wandbruntime‚9868  _runtime‚9868àŸU[€&Õ 2loss/SRqqt3/assignment_loss‚1.2608399249885867 1loss/FBt4/assignment_loss‚0.14534768836437398 /loss/FBt2/detection_loss‚2.1742818572304468 0loss/FRt4/assignment_loss‚0.9729492879576154 1loss/FBt3/assignment_loss‚0.14670764018654836 /loss/FRt1/detection_loss‚2.2441925352173193 2loss/FBt2/assignment_loss‚0.029788038972500628 .loss/FRt2/detection_loss‚2.338911561643313 /loss/FRt3/detection_loss‚2.8293076852957406 _step‚760 .loss/FRt1/assignment_loss‚3.42903562443597 2loss/SRqqt1/assignment_loss‚1.3962910247345766 epoch‚7 % _timestamp‚1.7884977686349258e+09 /loss/FRt3/assignment_loss‚3.040718448162079 trainer/global_step‚9249 0loss/FBt1/assignment_loss‚0.1549290007775177 0loss/SRqqt2/detection_loss‚2.570991337299347 %loss/total_loss‚39.25493485354045 2loss/SRqqt2/assignment_loss‚1.5291083271686847 2loss/SRqqt4/assignment_loss‚0.7494375670181685 /loss/FRt4/detection_loss‚1.4531059982772503 0loss/SRqqt3/detection_loss‚1.890787343971613 /loss/FBt4/detection_loss‚0.5009216508522296 _runtime‚9868.275081532 0loss/SRqqt1/detection_loss‚2.164481032888095 1loss/SRqqt4/detection_loss‚0.9089705145672748 /loss/FBt1/detection_loss‚2.3426911937338963 /loss/FBt3/detection_loss‚2.4681813791394234 /loss/FRt2/assignment_loss‚2.512958190657876ßU¨M`&Ú .loss/FRt1/assignment_loss‚3.42903562443597 2loss/SRqqt1/assignment_loss‚1.3962910247345766 epoch‚7 % _timestamp‚1.7884977686349258e+09 /loss/FRt3/assignment_loss‚3.040718448162079 trainer/global_step‚9249 0loss/FBt1/assignment_loss‚0.1549290007775177 0loss/SRqqt2/detection_loss‚2.570991337299347 %loss/total_loss‚39.25493485354045 2loss/SRqqt2/assignment_loss‚1.5291083271686847 2loss/SRqqt4/assignment_loss‚0.7494375670181685 /loss/FRt4/detection_loss‚1.4531059982772503 0loss/SRqqt3/detection_loss‚1.890787343971613 /loss/FBt4/detection_loss‚0.5009216508522296 _runtime‚9868.275081532 0loss/SRqqt1/detection_loss‚2.164481032888095 1loss/SRqqt4/detection_loss‚0.9089705145672748 /loss/FBt1/detection_loss‚2.3426911937338963 /loss/FBt3/detection_loss‚2.4681813791394234 /loss/FRt2/assignment_loss‚2.512958190657876 2loss/SRqqt3/assignment_loss‚1.2608399249885867 1loss/FBt4/assignment_loss‚0.14534768836437398 /loss/FBt2/detection_loss‚2.1742818572304468 0loss/FRt4/assignment_loss‚0.9729492879576154 1loss/FBt3/assignment_loss‚0.14670764018654836 /loss/FRt1/detection_loss‚2.2441925352173193 2loss/FBt2/assignment_loss‚0.029788038972500628 .loss/FRt2/detection_loss‚2.338911561643313 /loss/FRt3/detection_loss‚2.8293076852957406 _step‚760øqG®†œ‚&‚(:‘ êžéÔ¯þÞ± gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.72' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.544000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04üÞVyƒ&‚(:î êžéÔŠÈà³ disk.dm-1.out‚ 624.60546875 disk.drbd1260.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 31230.0703125 disk.dm-0.out‚234724.1015625 disk.drbd1260.out‚1.390625 disk.sda.in‚0 disk.sda.out‚ 624.60546875 disk.sda2.out‚0# disk.nvme0n1.out‚234724.1015625# proc.memory.rssMB‚ 2949.64453125* proc.memory.percent‚18.003201484680176 cpu‚25.289699772163264% disk./.usageGB‚1175.2284507751465 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 31230.0703125$ memory_percent‚93.17331314086914' proc.memory.availableMB‚ 1118.484375 proc.cpu.threads‚17 network.recv‚ 3.249404e+06 disk.sda1.in‚0 disk.sda1.out‚0% disk.nvme0c0n1.out‚234724.1015625! disk.nvme0n1.in‚ 31230.0703125 network.sent‚ 2.3372017e+07) disk./.usagePercent‚78.87056741387005 disk.sda3.out‚ 624.60546875@À¬‚‘„&‚(:† ùžéÔƒ´Ã­ gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.899 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„nu…&‚(:ö ùžéÔÿýЯ) disk./.usagePercent‚78.80038346037314# disk./.usageGB‚1174.18265914917 disk.drbd1260.out‚ 1.39453125 disk.sda2.in‚0( disk.nvme0c0n1.out‚235293.2705078125# disk.dm-0.out‚235293.2705078125$ memory_percent‚93.170285229nGÌs491455) proc.memory.availableMB‚ 1118.98046875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 31361.5546875 network.sent‚ 2.3480228e+07 network.recv‚ 3.255282e+06 disk.sda3.out‚ 625.0078125& disk.nvme0n1.out‚235293.2705078125# proc.memory.rssMB‚ 2949.64453125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.out‚ 625.0078125 disk.sda.in‚0 disk.sda.out‚ 625.0078125# disk.nvme0c0n1.in‚ 31361.5546875! disk.nvme0n1.in‚ 31361.5546875* proc.memory.percent‚18.003201484680176 cpu‚25.289435820045682x²Õ’†&‚(:‡ ˆŸéÔ´ÉŽ® gpu.0.gpu‚49.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.066' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§£™„…‡&‚(:ú ˆŸéÔ©Šñ®# proc.memory.rssMB‚ 2949.64453125 proc.cpu.threads‚17% disk./.usageGB‚1174.1587867736816 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 31361.5546875$ disk.dm-0.out‚235664.34716796875 network.sent‚ 2.3486192e+07 disk.drbd1260.out‚ 1.39453125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚235664.34716796875$ memory_percent‚93.17052364349365) proc.memory.availableMB‚ 1118.94140625 disk.sda3.out‚ 625.4296875 disk.dm-1.out‚ 625.4296875 disk.sda.in‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚235664.34716796875! disk.nvme0n1.in‚ 31361.5546875* proc.memory.percent‚18.003201484680176 cpu‚25.28918010929959 network.recv‚ 3.259666e+06) disk./.usagePercent‚78.79878136522387 disk.dm-1.in‚0 disk.sda.out‚ 625.4296875# disk.nvme0c0n1.in‚ 31361.5546875:KÉ’ˆ&‚(:‡ —ŸéÔ¨þ¶® gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚124.381' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8762 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/¦õ‰&‚(:ô —ŸéÔ‡úѯ disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚236049.7666015625! disk.nvme0n1.in‚ 31361.5546875& disk.nvme0n1.out‚236049.7666015625( proc.memory.availableMB‚ 1118.9453125 network.sent‚ 2.3491338e+07 network.recv‚ 3.26336e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 625.8515625 disk.sda.in‚0# disk.nvme0c0n1.in‚ 31361.5546875 disk.sda3.in‚0* proc.memory.percent‚18.003201484680176$ disk./.usageGB‚1173.101276397705 disk.drbd1260.out‚ 1.3984375 disk.sda.out‚ 625.8515625# proc.memory.rssMB‚ 2949.64453125 cpu‚25.288842236690734 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 31361.5546875# disk.dm-0.out‚236049.7666015625 disk.sda3.out‚ 625.8515625$ memory_percent‚93.17049980163574 proc.cpu.threads‚17) disk./.usagePercent‚78.72781095658178 disk.sda1.in‚0Ù·Ãê2Š&- _wandbruntime‚9915  _runtime‚9915B* OÉ‹&à % _timestamp‚1.7884978162764175e+09 _runtime‚9915.916971602 _step‚761 ' lr-AdamW/pg1‚0.00012180084776010037 ' lr-AdamW/pg2‚0.00012180084776010037 trainer/global_step‚92990* aÎŒ&È _step‚761 ' lr-AdamW/pg1‚0.00012180084776010037 ' lr-AdamW/pg2‚0.00012180084776010037 trainer/global_step‚9299 % _timestamp‚1.7884978162764175e+09 _runtime‚9915.916971602ù"v©ð2&- _wandbruntime‚9915  _runtime‚9915¡sð7û'Ž&õO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚642606079115628 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 $ _timestamp‚1.788497816277252e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206199386 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚721475434675200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 _step‚762 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚659233568296448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9959778732032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9959588784128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144950723 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚633088860334128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551623226368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93251997 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 368857308 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9551623226368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 368880326 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚731435023459328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649681963073024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206222402 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚633088843294768 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206199386 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9552395871232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649681945070080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144950187 trainer/global_step‚9299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162657924 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649681945070080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649681963073024 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162657924 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚721475458615296 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162657922 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51698192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚659234358944256 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552395871232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9517235820860 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚659234358944256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93251995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚659233568296448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536ÇÅÉ(&úO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162657924 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649681945070080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649681963073024 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162657924 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚721475458615296 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162657922 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51698192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚659234358944256 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552395871232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9517235820860 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚659234358944256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93251995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚659233568296448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚642606079115628 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 $ _timestamp‚1.788497816277252e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206199386 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚721475434675200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 _step‚762 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚659233568296448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9959778732032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9959588784128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144950723 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚633088860334128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551623226368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93251997 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 368857308 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9551623226368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 368880326 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚731435023459328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649681963073024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206222402 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚633088843294768 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206199386 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9552395871232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649681945070080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144950187 trainer/global_step‚9299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39ú­Bú 2&- _wandbruntime‚9916  _runtime‚9916ÄØx:â ‘&€N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162674796 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144965693 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚562 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51703488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 368895893 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162674798 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9518236960738 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9553400630272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚659302272981504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368918937 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚731511417839616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚649749663012352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚721550806958080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9519007609358 _runtime‚9916.778987922 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93261643 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚649749645009408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚731511642200064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 trainer/global_step‚9299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206221097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9960634821632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144965131 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206221097 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚659303063642624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9553400630272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚´ñù$642673934379246 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649749663012352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9960835241984 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93261645 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584èïÔ '’&…N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162674796 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144965693 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚562 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51703488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 368895893 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162674798 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9518236960738 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9553400630272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚659302272981504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368918937 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚731511417839616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚649749663012352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚721550806958080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9519007609358 _runtime‚9916.778987922 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93261643 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚649749645009408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚731511642200064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 trainer/global_step‚9299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206221097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9960634821632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144965131 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206221097 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚659303063642624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9553400630272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚642673934379246 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649749663012352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9960835241984 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93261645 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328ûÙ5cj2“&- _wandbruntime‚9916  _runtime‚9916õ‘:q[”&Õ /loss/FBt4/detection_loss‚0.3971647902825763 1loss/FBt3/assignment_loss‚0.18688066939001877 epoch‚7 trainer/global_step‚9299 _step‚764 %loss/total_loss‚40.24642672527587 0loss/FRt1/assignment_loss‚3.6346464493335824 0loss/FRt3/assignment_loss‚2.1641989818641116 1loss/SRqqt3/detection_loss‚1.7229870720207692 1loss/SRqqt4/detection_loss‚0.7812981927717054 /loss/FRt2/assignment_loss‚2.876894278772946 0loss/FRt4/assignment_loss‚1.0694236335620073 2loss/SRqqt1/assignment_loss‚1.1556459423154593 /loss/FRt1/detection_loss‚2.5474295929456368 /loss/FRt4/detection_loss‚1.3391109249016768 .loss/FBt1/detection_loss‚2.753203617201911 _runtime‚9916.780607254 /loss/FRt2/detection_loss‚2.7262764820071768 2loss/SRqqt2/assignment_loss‚1.6650204613412682 2loss/SRqqt3/assignment_loss‚1.0065213401429356 2loss/SRqqt4/assignment_loss‚0.9824149063998112 1loss/FBt1/assignment_loss‚0.13587051887901916 /loss/SRqqt2/detection_loss‚2.62287406858645 $ _timestamp‚1.788497817140473e+09 1loss/FBt4/assignment_loss‚0.11726825925411012 0loss/SRqqt1/detection_loss‚2.402917837103208 0loss/FBt2/assignment_loss‚0.7139590559242833 /loss/FRt3/detection_loss‚2.4663446036034395 /loss/FBt2/detection_loss‚2.8645370841026305 .loss/FBt3/detection_loss‚1.913537962569131±¾Qw`•&Ú 0loss/FBt2/assignment_loss‚0.7139590559242833 /loss/FRt3/detection_loss‚2.4663446036034395 /loss/FBt2/detection_loss‚2.8645370841026305 .loss/FBt3/detection_loss‚1.913537962569131 /loss/FBt4/detection_loss‚0.3971647902825763 1loss/FBt3/assignment_loss‚0.18688066939001877 epoch‚7 trainer/global_step‚9299 _step‚764 %loss/total_loss‚40.24642672527587 0loss/FRt1/assignment_loss‚3.6346464493335824 0loss/FRt3/assignment_loss‚2.1641989818641116 1loss/SRqqt3/detection_loss‚1.7229870720207692 1loss/SRqqt4/detection_loss‚0.7812981927717054 /loss/FRt2/assignment_loss‚2.876894278772946 0loss/FRt4/assignment_loss‚1.0694236335620073 2loss/SRqqt1/assignment_loss‚1.1556459423154593 /loss/FRt1/detection_loss‚2.5474295929456368 /loss/FRt4/detection_loss‚1.3391109249016768 .loss/FBt1/detection_loss‚2.753203617201911 _runtime‚9916.780607254 /loss/FRt2/detection_loss‚2.7262764820071768 2loss/SRqqt2/assignment_loss‚1.6650204613412682 2loss/SRqqt3/assignment_loss‚1.0065213401429356 2loss/SRqqt4/assignment_loss‚0.9824149063998112 1loss/FBt1/assignment_loss‚0.13587051887901916 /loss/SRqqt2/detection_loss‚2.62287406858645 $ _timestamp‚1.788497817140473e+09 1loss/FBt4/assignment_loss‚0.11726825925411012 0loss/SRqqt1/detection_loss‚2.402917837103208üÜ3‹3’–&‚(:‡ ¦ŸéÔŠùç¯ gpu.0.gpu‚24.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0412 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ká…ôo—&‚(:ä ¦ŸéÔ·ªí° disk.drbd1260.out‚ 1.3984375 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 31361.5546875) proc.memory.availableMB‚ 1119.16796875 disk.sda3.out‚626.25 disk.sda.out‚626.25 disk.sda1.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2949.64453125 cpu‚25.288527910006653 network.sent‚ 2.3509451e+07 network.recv‚ 3.26677e+06 disk.sda3.in‚0 disk.dm-1.out‚626.25 disk.sda2.in‚0( disk.nvme0c0n1.out‚236470.6259765625$ memory_percent‚93.16914081573486# disk./.usageGB‚1172.01314163208 disk.drbd1260.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 31361.5546875& disk.nvme0n1.out‚236470.6259765625* proc.memory.percent‚18.003201484680176 proc.cpu.threads‚17( disk./.usagePercent‚78.6547853194548 disk.dm-0.in‚ 31361.5546875# disk.dm-0.out‚236470.6259765625 disk.dm-1.in‚0<Ðý#˜&‚(:… µŸéÔ¨Øý® gpu.0.gpu‚49.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.52' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆürøx™&‚(:í µŸéÔ˼á±( disk.nvme0c0n1.out‚236639.8095703125$ memory_percent‚93.17066669464111 cpu‚25.288241055720437 proc.cpu.threads‚17% disk./.usageGB‚1172.0131072998047 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.40234375 disk.sda.in‚0 disk.sda2.out‚0# disk.dm-0.out‚236639.8095703125# proc.memory.rssMB‚ 2949.64453125 network.recv‚ 3.273576e+06' disk./.usagePercent‚78.654783015387 disk.dm-1.out‚ 626.6484375 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚18.003201484680176 network.sent‚ 2.3605615e+07 disk.sda1.in‚0 disk.nvme0n1.in‚ 31500.6875& disk.nvme0n1.out‚236639.8095703125 disk.dm-0.in‚ 31500.6875) proc.memory.availableMB‚ 1118.91796875 disk.sda3.in‚0 disk.sda3.out‚ 626.6484375 disk.sda.out‚ 626.6484375 disk.nvme0c0n1.in‚ 31500.6875ü±”š&‚(:… ÄŸéÔǸë­ gpu.0.gpu‚50.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.01' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)…–´{›&‚(:ð ÄŸéÔÁ©ß¯( disk.nvme0c0n1.out‚236809.4658203125# disk.dm-0.out‚236809.4658203125$ memory_percent‚93.17078590393066( proc.memory.availableMB‚ 1118.8984375 cpu‚25.287945419012978 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 626.98828125 disk.drbd1260.out‚ 1.40234375& disk.nvme0n1.out‚236809.4658203125 disk.dm-0.in‚ 31500.6875* proc.memory.percent‚18.003201484680176 proc.cpu.threads‚17 network.sent‚ 2.3611499e+07 network.recv‚ 3.278145e+06$ disk./.usageGB‚1171.902988433838 disk.sda1.in‚0 disk.nvme0n1.in‚ 31500.6875# proc.memory.rssMB‚ 2949.64453125 disk.sda1.out‚0 disk.sda3.out‚ 626.98828125 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 626.98828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 31500.6875) disk./.usagePercent‚78.64739284589608ã¦È¾2œ&- _wandbruntime‚9962  _runtime‚9962yÉ&à ' lr-AdamW/pg1‚0.00011770426500871233 ' lr-AdamW/pg2‚0.00011770426500871233 trainer/global_step‚9349 % _timestamp‚1.7884978624103127e+09 _runtime‚9962.050601944 _step‚765öv]©Îž&È trainer/global_step‚9349 % _timestamp‚1.7884978624103127e+09 _runtime‚9962.050601944 _step‚765 ' lr-AdamW/pg1‚0.00011770426500871233 ' lr-AdamW/pg2‚0.00011770426500871233ý—ÑQô2Ÿ&- _wandbruntime‚9962  _runtime‚9962Ï_½Êþ' &øO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9602647571456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163501624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚653066942036480 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10011974331904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚653066960039424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9567306405968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 207311304 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚653066960039424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145699392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93734395 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚636392182122128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9349 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚662669607610880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚653066942036480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 370812928 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10012170571264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9568077054380 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 _step‚766 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163501622 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163501622 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚662668816963072 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚636392165082768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884978624113054e+09 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚662669607610880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51965535 _runtime‚9962.051723217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207288288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚735254826146304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207311304ÊøÜÿ`¡&ýO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚636392165082768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884978624113054e+09 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMoniq¼£tor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚662669607610880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51965535 _runtime‚9962.051723217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207288288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚735254826146304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207311304 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9602647571456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163501624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚653066942036480 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10011974331904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚653066960039424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9567306405968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 207311304 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚653066960039424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145699392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93734395 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚636392182122128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9349 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚662669607610880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚653066942036480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 370812928 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10012170571264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9568077054380 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 _step‚766 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163501622 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163501622 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚662668816963072 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0þÓG¬ø2¢&- _wandbruntime‚9962  _runtime‚9962ñÇÅW'£&‚N YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9603652330496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51970192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚735331442561536 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51970710 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚646027326752354 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚9962.920623147 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚725318200157184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163518498 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚636458231518528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163518496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163518496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145714235 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚653134659978752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93744045 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚662738312309248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚653134641975808 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 207333041 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207309999 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 370851539 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207333041 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚9349 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 % _timestamp‚1.7884978632799897e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚653134659978752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9568307545846 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93744043 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚636458248557888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 207309999 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9603652330496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚735331220298240 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163518498 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10013218464256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚646026539064374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352~EP '¤&‡N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 207333041 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207309999 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 370851539 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207333041 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚9349 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 % _timestamp‚1.7884978632799897e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚653134659978752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9568307545846 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93744043 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚636458248557888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 207309999 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9603652330496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚735331220298240 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163518498 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10013218464256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚646026539064374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9603652330496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51970192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚735331442561536 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51970710 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚646027326752354 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚9962.920623147 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚725318200157184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163518498 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚636458231518528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163518496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163518496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145714235 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚653134659978752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93744045 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚662738312309248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚653134641975808 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075ÿ(†Æâ2¥&- _wandbruntime‚9962  _runtime‚9962Ù<— Z¦&Ô 1loss/FBt2/assignment_loss‚0.07348260683584858 .loss/FRt1/detection_loss‚2.188323857299253 0loss/SRqqt1/detection_loss‚2.116476585183825 0loss/SRqqt3/detection_loss‚2.317998836437861 1loss/SRqqt4/assignment_loss‚0.942921847697686 /loss/FRt3/detection_loss‚2.8695342163244884 .loss/FBt3/detection_loss‚2.002001237869263 &loss/total_loss‚39.215372815762294 epoch‚7 trainer/global_step‚9349 0loss/FRt3/assignment_loss‚2.8344280223051705 1loss/SRqqt2/assignment_loss‚1.232920132045235 1loss/FBt1/assignment_loss‚0.17874072088527543 0loss/FBt3/assignment_loss‚0.3137439731467794 /loss/FRt4/detection_loss‚1.3132606281691954 /loss/FBt2/detection_loss‚2.6655223277898936 0loss/FBt4/detection_loss‚0.46487188063819257 _runtime‚9962.922693518 $ _timestamp‚1.788497863282406e+09 2loss/SRqqt3/assignment_loss‚1.2835361930231253 /loss/FRt2/detection_loss‚2.6771843602308185 /loss/FRt1/assignment_loss‚2.649885784177219 0loss/FRt2/assignment_loss‚3.4188942333747603 1loss/SRqqt4/detection_loss‚0.7189802393091448 _step‚768 0loss/FRt4/assignment_loss‚1.2240611521734133 2loss/SRqqt1/assignment_loss‚1.1379890067236764 0loss/FBt4/assignment_loss‚0.1137512312649782 0loss/SRqqt2/detection_loss‚2.266399013144629 /loss/FBt1/detection_loss‚2.2104647297125597òŒaþ_§&Ù $ _timestamp‚1.788497863282406e+09 2loss/SRqqt3/assignment_loss‚1.2835361930231253 /loss/FRt2/detection_loss‚2.6771843602308185 /loss/FRt1/assignment_loss‚2.649885784177219 0loss/FRt2/assignment_loss‚3.4188942333747603 1loss/SRqqt4/detection_loss‚0.7189802393091448 _step‚768 0loss/FRt4/assignment_loss‚1.2240611521734133 2loss/SRqqt1/assignment_loss‚1.1379890067236764 0loss/FBt4/assignment_loss‚0.1137512312649782 0loss/SRqqt2/detection_loss‚2.266399013144629 /loss/FBt1/detection_loss‚2.2104647297125597 1loss/FBt2/assignment_loss‚0.07348260683584858 .loss/FRt1/detection_loss‚2.188323857299253 0loss/SRqqt1/detection_loss‚2.116476585183825 0loss/SRqqt3/detection_loss‚2.317998836437861 1loss/SRqqt4/assignment_loss‚0.942921847697686 /loss/FRt3/detection_loss‚2.8695342163244884 .loss/FBt3/detection_loss‚2.002001237869263 &loss/total_loss‚39.215372815762294 epoch‚7 trainer/global_step‚9349 0loss/FRt3/assignment_loss‚2.8344280223051705 1loss/SRqqt2/assignment_loss‚1.232920132045235 1loss/FBt1/assignment_loss‚0.17874072088527543 0loss/FBt3/assignment_loss‚0.3137439731467794 /loss/FRt4/detection_loss‚1.3132606281691954 /loss/FBt2/detection_loss‚2.6655223277898936 0loss/FBt4/detection_loss‚0.46487188063819257 _runtime‚9962.922693518€â`O‘¨&‚(:† ÓŸéÔ…¼º® gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.401 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž‹†ln©&‚(:ã ÓŸéÔ”Ç÷° disk.sda3.out‚ 627.40234375 disk.dm-1.out‚ 627.40234375 disk.sda.out‚ 627.40234375 disk.sda2.in‚0 disk.dm-0.in‚ 31500.6875 disk.dm-0.out‚236990.3984375# proc.memory.rssMB‚ 2949.64453125 network.recv‚ 3.281736e+06 disk.sda3.in‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚236990.3984375 disk.nvme0n1.in‚ 31500.6875( proc.memory.availableMB‚ 1118.8828125 disk.drbd1260.out‚1.40625 disk.sda.in‚0# memory_percent‚93.1708812713623* proc.memory.percent‚18.003201484680176 cpu‚25.287609134194053 proc.cpu.threads‚17 network.sent‚ 2.3616595e+07) disk./.usagePercent‚78.64732935602758 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 31500.6875# disk.nvme0n1.out‚236990.3984375$ disk./.usageGB‚1171.902042388916ê.CRª&‚(:… âŸéÔŽšÊ­ gpu.0.gpu‚38.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚126.49' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ5áÑn«&‚(:ã âŸéÔäŸï¯ disk.sda3.out‚ 627.73828125 disk.drbd1260.out‚1.40625 disk.nvme0n1.in‚ 31500.6875# memory_percent‚93.1708812713623# proc.memory.rssMB‚ 2949.64453125* proc.memory.percent‚18.003201484680176 network.sent‚ 2.3724605e+07 network.recv‚ 3.287537e+06) disk./.usagePercent‚78.65912874328073% disk./.usageGB‚1172.0778617858887 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0% disk.nvme0c0n1.out‚237272.8359375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 627.73828125 disk.sda2.in‚0# disk.nvme0n1.out‚237272.8359375 disk.dm-0.in‚ 31500.6875 disk.dm-0.out‚237272.8359375 disk.sda3.in‚0 disk.dm-1.out‚ 627.73828125 disk.nvme0c0n1.in‚ 31500.6875( proc.memory.availableMB‚ 1118.8828125 cpu‚25.28731041922959 proc.cpu.threads‚17Õ†Ò¬&‚(:’ ñŸéÔ¿ÞÑ­ gpu.0.gpu‚42.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.284' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.256799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰^ Ï6­&‚(:ý ñŸéÔö“а( proc.memory.availableMB‚ 1118.8828125* proc.memory.percent‚18.003201484680176 cpu‚25.28700319847797 proc.cpu.threads‚17) disk./.usagePercent‚78.65910416655744% disk./.usageGB‚1172.0774955749512 disk.sda3.out‚ 628.15234375 disk.dm-1.out‚ 628.15234375—:ÅR disk.sda1.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚237363.35009765625# proc.memory.rssMB‚ 2949.64453125 network.recv‚ 3.29216e+06 disk.sda.in‚0' disk.nvme0n1.out‚237363.35009765625 disk.dm-0.in‚31698.48046875 network.sent‚ 2.3732795e+07 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.41015625 disk.sda.out‚ 628.15234375 disk.sda2.in‚0$ disk.nvme0c0n1.in‚31698.48046875" disk.nvme0n1.in‚31698.48046875$ disk.dm-0.out‚237363.35009765625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0# memory_percent‚93.1708812713623Fâµ4®&/ _wandbruntime‚10009  _runtime‚10009Õh’Sʯ&Ä % _timestamp‚1.7884979099854388e+09 _runtime‚10009.625983195 _step‚769 ' lr-AdamW/pg1‚0.00011366570389887523 ' lr-AdamW/pg2‚0.00011366570389887523 trainer/global_step‚9399$8jÏ°&É % _timestamp‚1.7884979099854388e+09 _runtime‚10009.625983195 _step‚769 ' lr-AdamW/pg1‚0.00011366570389887523 ' lr-AdamW/pg2‚0.00011366570389887523 trainer/global_step‚9399Z°(ë4±&/ _wandbruntime‚10009  _runtime‚100091`b&ÿ'²&ùO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚666104065629696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 208400206 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚656451957005824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9652126626816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9652899271680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚649313651549616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚639695486870768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9618147639488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10064497373184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9652899271680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚649312863861844 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9617376991076 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52231806 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚656451939002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 % _timestamp‚1.7884979099867325e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚656451939002880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10064689418240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164345322 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚666104856277504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚666104856277504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208377190 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9652126626816 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164345322 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚666104065629696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94216797 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164345324 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚739074766326784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚10009.627295067 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚639695503910128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 372745530 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146448603 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146448054 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52231259 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚739074982311936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208400206 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚729010268953600 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208377190 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 372745530 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146448603 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146448054 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52231259 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚739074982311936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208400206 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚729010268953600 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208377190 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚666104065629696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 208400206 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚656451957005824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9652126626816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9652899271680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚649313651549616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚639695486870768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9618147639488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10064497373184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9652899271680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚649312863861844 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9617376991076 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52231806 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚656451939002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 % _timestamp‚1.7884979099867325e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚656451939002880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10064689418240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164345322 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚666104856277504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚666104856277504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208377190 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9652126626816 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164345322 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚666104065629696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94216797 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164345324 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚739074766326784‚ÊæÈr4´&/ _wandbruntime‚10010  _runtime‚100107±{Ò 'µ&ƒN RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164362198 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 372761097 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52236525 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9653904030720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 372784141 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164362196 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164362198 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208421943 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9653131372544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9653904030720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52237060 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10065740023808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚649380719125462 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚656519638942208 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146462968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 trainer/global_step‚9399 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 372784141 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9618378130954 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146463505 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10065541700608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚639761570345888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚639761553306528©–V'¶&ˆN UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9653131372544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚649379931437482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164362196 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 bODeviceStatsBK³ç"Monitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10065740023808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚649380719125462 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚656519638942208 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146462968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 trainer/global_step‚9399 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 372784141 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9618378130954 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146463505 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10065541700608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚639761570345888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚639761553306528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164362198 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 372761097 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52236525 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9653904030720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 372784141 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164362196 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 164362198 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208421943 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9653131372544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9653904030720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52237060 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 Bôú÷4·&/ _wandbruntime‚10010  _runtime‚10010Ù‚Ö~\¸&Ö trainer/global_step‚9399 0loss/FBt3/assignment_loss‚0.2813900082171707 0loss/FRt2/assignment_loss‚2.7348705424202815 0loss/FBt1/assignment_loss‚0.1398591239594344 .loss/FRt3/detection_loss‚2.586713789900144 0loss/SRqqt2/detection_loss‚2.475137288229806 /loss/FRt1/assignment_loss‚2.991678597071232 0loss/FRt3/assignment_loss‚3.1208260993162793 0loss/FRt4/assignment_loss‚1.3084590062499046 2loss/SRqqt3/assignment_loss‚1.2025413869747095 1loss/SRqqt1/detection_loss‚2.3717945031821728 /loss/FBt4/detection_loss‚0.5350119820116864 epoch‚7 _step‚772 /loss/FRt2/detection_loss‚2.3915357054441757 1loss/SRqqt3/detection_loss‚1.5654774429276586 % _timestamp‚1.7884979109725583e+09 2loss/FBt4/assignment_loss‚0.053205458167432856 1loss/SRqqt2/assignment_loss‚1.671268993004092 1loss/FBt2/assignment_loss‚0.37970933173993704 .loss/FBt1/detection_loss‚2.181620711630041 .loss/FBt3/detection_loss‚1.556482425757817 1loss/SRqqt1/assignment_loss‚1.397197034675628 /loss/FRt1/detection_loss‚2.4334752213878508 /loss/FRt4/detection_loss‚1.3215016116543363 /loss/FBt2/detection_loss‚2.4034607960627627 %loss/total_loss‚38.69062225226972 _runtime‚10010.613069961 2loss/SRqqt4/assignment_loss‚0.9348219498911371 1loss/SRqqt4/detection_loss‚0.6525832423940301; Ea¹&Û /loss/FRt4/detection_loss‚1.3215016116543363 /loss/FBt2/detection_loss‚2.4034607960627627 %loss/total_loss‚38.69062225226972 _runtime‚10010.613069961 2loss/SRqqt4/assignment_loss‚0.9348219498911371 1loss/SRqqt4/detection_loss‚0.6525832423940301 trainer/global_step‚9399 0loss/FBt3/assignment_loss‚0.2813900082171707 0loss/FRt2/assignment_loss‚2.7348705424202815 0loss/FBt1/assignment_loss‚0.1398591239594344 .loss/FRt3/detection_loss‚2.586713789900144 0loss/SRqqt2/detection_loss‚2.475137288229806 /loss/FRt1/assignment_loss‚2.991678597071232 0loss/FRt3/assignment_loss‚3.1208260993162793 0loss/FRt4/assignment_loss‚1.3084590062499046 2loss/SRqqt3/assignment_loss‚1.2025413869747095 1loss/SRqqt1/detection_loss‚2.3717945031821728 /loss/FBt4/detection_loss‚0.5350119820116864 epoch‚7 _step‚772 /loss/FRt2/detection_loss‚2.3915357054441757 1loss/SRqqt3/detection_loss‚1.5654774429276586 % _timestamp‚1.7884979109725583e+09 2loss/FBt4/assignment_loss‚0.053205458167432856 1loss/SRqqt2/assignment_loss‚1.671268993004092 1loss/FBt2/assignment_loss‚0.37970933173993704 .loss/FBt1/detection_loss‚2.181620711630041 .loss/FBt3/detection_loss‚1.556482425757817 1loss/SRqqt1/assignment_loss‚1.397197034675628 /loss/FRt1/detection_loss‚2.4334752213878508„Å=Ldœº&‚(:‘ € éÔ°Ã’¯ gpu.0.gpu‚48.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.35' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.869999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ωþÓŠ»&‚(:ÿ € éÔ‰èð°% disk./.usageGB‚1172.0774574279785 disk.dm-1.in‚0 disk.sda.out‚ 628.61328125 disk.sda2.in‚0$ disk.nvme0c0n1.in‚31698.48046875) disk.nvme0c0n1.out‚237508.11962890625' disk.nvme0n1.out‚237508.11962890625 disk.sda3.in‚0 network.sent‚ 2.3736738e+07( disk./.usagePercent‚78.6591016064821 disk.sda1.in‚0 disk.dm-0.in‚31698.48046875$ disk.dm-0.out‚237508.11962890625 disk.sda3.out‚ 628.61328125) proc.memory.availableMB‚ 1118.56640625 cpu‚25.28677360685678 network.recv‚ 3.296585e+06 disk.dm-1.out‚ 628.61328125 disk.drbd1260.out‚ 1.41015625 disk.sda1.out‚0" disk.nvme0n1.in‚31698.48046875# proc.memory.rssMB‚ 2949.64453125 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚93.17281246185303* proc.memory.percent‚18.003201484680176 f8-’¼&‚(:‡  éÔíû”° gpu.0.gpu‚40.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚146.096' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.2192 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬Ak†½&‚(:û  éÔäÛã´$ memory_percent‚93.17288398742676* proc.memory.percent‚18.003201484680176 cpu‚25.286550500269964 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 629.19921875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 629.19921875( disk.nvme0c0n1.out‚237684.0029296875 network.sent‚ 2.384473e+07 network.recv‚ 3.302107e+06) disk./.usagePercent‚78.64683423745636 disk.drbd1260.out‚ 1.4140625" disk.nvme0n1.in‚31698.48046875& disk.nvme0n1.out‚237684.0029296875 disk.dm-0.in‚31698.48046875( proc.memory.availableMB‚ 1118.5546875 disk.dm-1.out‚ 629.19921875 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚31698.48046875# proc.memory.rssMB‚ 2949.64453125% disk./.usageGB‚1171.8946647644043 disk.sda1.out‚0# disk.dm-0.out‚237684.0029296875‡["œ¾&‚(:‘ ž éÔþ´é® gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.536' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.907200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÏ>‚¿&‚(:÷ ž éÔ´´Š± network.sent‚ 2.3849726e+07 network.recv‚ 3.305614e+06( disk./.usagePercent‚78.6459784042694 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.4140625$ disk.nvme0c0n1.in‚31698.48046875& disk.nvme0n1.out‚237952.4052734375 disk.dm-0.in‚31698.48046875% disk./.usageGB‚1171.8819122314453 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚237952.4052734375# proc.memory.rssMB‚ 2949.64453125 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 629.61328125" disk.nvme0n1.in‚31698.48046875$ memory_percent‚93.17131042480469% proc.memory.availableMB‚ 1118.8125* proc.memory.percent‚18.003201484680176 cpu‚25.28629351509044 disk.sda3.out‚ 629.61328125 disk.dm-1.out‚ 629.61328125( disk.nvme0c0n1.out‚237952.4052734375 proc.cpu.threads‚17­E$~4À&/ _wandbruntime‚10059  _runtime‚10059ðäÄÊÁ&Ä ' lr-AdamW/pg1‚0.00010968600606067134 ' lr-AdamW/pg2‚0.00010968600606067134 trainer/global_step‚9449 % _timestamp‚1.7884979601852348e+09 _runtime‚10059.825537315 _step‚773üf ÏÂ&É ' lr-AdamW/pg1‚0.00010968600606067134 ' lr-AdamW/pg2‚0.00010968600606067134 trainer/global_step‚9449 % _timestamp‚1.7884979601852348e+09 _runtime‚10059.825537315 _step‚773…YWû4Ã&/ _wandbruntime‚10059  _runtime‚10059Æ\9Àÿ'Ä&ùO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚652667043922724 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 trainer/global_step‚9449 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚659836935969280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209466092 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 % _timestamp‚1.7884979601861968e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94699197 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165189022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52497218 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9702378327040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚659836953972224 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374655114 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165189022 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9702378327040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚669539314296320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209489108 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚669540104944128 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10116867771392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚732777686092800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚642998825698128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚669540104944128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165189024 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209466092 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚659836953972224 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209489108 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9703150971904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165189024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚642998808658768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94699195 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9667447576184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚659836935969280 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10117064010752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚774 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374678132 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚742894774043648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9703150971904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9668218224596 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚669539314296320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚732777710032896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚652666256234952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360¶Àf\Å&þO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374655114 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165189022 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9702378327040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚669539314296320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209489108 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚669540104944128 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10116867771392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚732777686092800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚642998825698128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚669540104944128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165189024 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209466092 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚659836953972224 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209489108 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9703150971904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165189024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚642998808658768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94699195 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9667447576184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚659836935969280 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10117064010752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _step‚774 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374678132 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚742894774043648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocaÖ<¨tions.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9703150971904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9668218224596 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚669539314296320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚732777710032896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚652666256234952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚652667043922724 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 trainer/global_step‚9449 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚659836935969280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209466092 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 % _timestamp‚1.7884979601861968e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94699197 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165189022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52497218 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9702378327040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚659836953972224†˪~4Æ&/ _wandbruntime‚10060  _runtime‚10060.=Šú 'Ç&ƒN \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9669219364682 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9704155730944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209510845 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374716743 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165205898 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚553 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374716743 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9703383072768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚551 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚643064892133888 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚669608809642496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374693699 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209487803 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94708843 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚742971172392960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚732853034435584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147211871 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10118114017280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10117919888384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209510845 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 _runtime‚10060.673083309 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚659904635908608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚732853058375680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9668448716062 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165205898 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _step‚775 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚652733323810590 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 % _timestamp‚1.7884979610324109e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9704155730944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165205896 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147211318 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374716743 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165205898 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚553 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374716743 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9703383072768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚551 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚643064892133888 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚669608809642496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374693699 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209487803 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94708843 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚742971172392960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚732853034435584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147211871 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10118114017280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10117919888384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209510845 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 _runtime‚10060.673083309 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚659904635908608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚732853058375680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9668448716062 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165205898 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _step‚775 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚652733323810590 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 % _timestamp‚1.7884979610324109e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9704155730944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165205896 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147211318 B³Ì&‚(:’ ­ éԃ͗° gpu.0.gpu‚35.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.397000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k­â³ˆÍ&‚(:ý ­ éÔÀ¼”± cpu‚25.28599301044787 proc.cpu.threads‚17( disk.nvme0c0n1.out‚238269.9931640625 network.sent‚ 2.3869567e+07 disk.drbd1260.out‚ 1.41796875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚238269.9931640625 disk.dm-1.out‚ 630.08984375 disk.sda.out‚ 630.08984375$ disk.nvme0c0n1.in‚31857.07421875# disk.dm-0.out‚238269.9931640625 disk.sda3.in‚0 disk.dm-1.in‚0$ memory_percent‚93.16999912261963) proc.memory.availableMB‚ 1119.02734375# proc.memory.rssMB‚ 2949.64453125 network.recv‚ 3.311124e+06% disk./.usageGB‚1172.1233863830566 disk.sda2.in‚0" disk.nvme0n1.in‚31857.07421875 disk.dm-0.in‚31857.07421875 disk.sda3.out‚ 630.08984375* proc.memory.percent‚18.003201484680176) disk./.usagePercent‚78.66218393719436 disk.drbd1260.in‚0 disk.sda1.out‚0¢Í¿jÎ&‚(:’ ¼ éÔ†ƒˆ­ gpu.0.gpu‚32.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.614' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.922800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RlÂË…Ï&‚(:ú ¼ éÔݲβ cpu‚25.285686498668746 network.sent‚ 2.3965504e+07 disk.sda.in‚0 disk.sda.out‚ 630.4609375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 630.4609375 disk.drbd1260.in‚0 network.recv‚ 3.317664e+06% disk./.usageGB‚1172.3929481506348$ disk.nvme0c0n1.in‚31857.07421875" disk.nvme0n1.in‚31857.07421875& disk.nvme0n1.out‚238541.0283203125# disk.dm-0.out‚238541.0283203125 disk.dm-1.in‚0) proc.memory.availableMB‚ 1118.28515625( disk./.usagePercent‚78.6802744535939 disk.sda3.in‚0 disk.dm-1.out‚ 630.4609375$ memory_percent‚93.17452907562256 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚238541.0283203125 disk.dm-0.in‚31857.07421875 disk.drbd1260.out‚ 1.41796875# proc.memory.rssMB‚ 2949.64453125* proc.memory.percent‚18.003201484680176÷¦»w’Ð&‚(:‡ Ë éÔÝÃÓ® gpu.0.gpu‚40.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨P$ï|Ñ&‚(:ñ Ë éÔ†–Å´ disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0& disk.nvme0c0n1.out‚238900.55859375 disk.dm-1.in‚0 disk.sda.out‚ 630.98828125$ memory_percent‚93.17302703857422& proc.memory.availableMB‚ 1118.53125# proc.memory.rssMB‚ 2949.64453125 cpu‚25.285451822631718 network.sent‚ 2.3972664e+07 network.recv‚ 3.321462e+06# disk./.usageGB‚1172.43404006958" disk.nvme0n1.in‚31857.07421875 disk.sda3.in‚0 disk.sda3.out‚ 630.98828125 disk.dm-1.out‚ 630.98828125 disk.sda1.in‚0) disk./.usagePercent‚78.68303216675274$ disk.nvme0c0n1.in‚31857.07421875$ disk.nvme0n1.out‚238900.55859375 disk.drbd1260.in‚0 disk.sda.in‚0* proc.memory.percent‚18.003201484680176 disk.sda2.in‚0 disk.dm-0.in‚31857.07421875! disk.dm-0.out‚238900.55859375 disk.drbd1260.out‚1.421875E¾{õ’Ò&‚(:‡ Ú éÔó¿È­ gpu.0.gpu‚37.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚33.0 gpu.0.powerWatts‚124.379' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8758 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰æÓ ~Ó&‚(:ó Ú éÔÒØ ° cpu‚25.285274788634524 disk.drbd1260.in‚0& disk.nvme0c0n1.out‚239195.83984375" disk.nvme0n1.in‚31857.07421875 disk.dm-0.in‚31857.07421875$ memory_percent‚93.17150115966797 proc.cpu.threads‚17% disk./.usageGB‚1172.5577850341797 disk.drbd1260.out‚1.421875 disk.sda1.in‚0 disk.sda1.out‚0! disk.dm-0.out‚239195.83984375# proc.memory.rssMB‚ 2949.64453125 network.sent‚ 2.3975826e+07 network.recv‚ 3.324792e+06) disk./.usagePercent‚78.69133679515596 disk.dm-1.out‚ 631.29296875 disk.sda.in‚0 disk.sda.out‚ 631.29296875 disk.sda2.in‚0$ disk.nvme0c0n1.in‚31857.07421875 disk.dm-1.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚239195.83984375 disk.sda3.in‚0 disk.sda3.out‚ 631.29296875& proc.memory.availableMB‚ 1118.78125* proc.memory.percent‚18.003201484680176À‹4Ô&/ _wandbruntime‚10116  _runtime‚10116€²+ÊÕ&Ä ' lr-AdamW/pg1‚0.00010576600085716527 ' lr-AdamW/pg2‚0.00010576600085716527 trainer/global_step‚9499 % _timestamp‚1.7884980165737936e+09 _runtime‚10116.215380995 _step‚777¹†¶ÓÏÖ&É _step‚777 ' lr-AdamW/pg1‚0.00010576600085716527 ' lr-AdamW/pg2‚0.00010576600085716527 trainer/global_step‚9499 % _timestamp‚1.7884980165737936e+09 _runtime‚10116.215380995‰4Ð-4×&/ _wandbruntime‚10116  _runtime‚10116ð>S-&Ø&ùO A7DeviceStatsMonitor.on_train_‡ÜÙ'batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147944608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52763013 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9753402672128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376610734 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚672975353610752 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9718288809704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚663221950938624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚746714760265216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210578010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚663221932935680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95181595 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚746714548474368 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9717518161292 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166032722 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147945139 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52763542 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚10116.217655369 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166032724 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚672975353610752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95181597 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚9499 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166032722 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚663221950938624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211790848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166032724 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚736545103232000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚656020436295832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚672974562962944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚663221932935680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10169633093120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚656019648608060 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210554994 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 210554994 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚736545127172096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884980165764835e+09“¸Í(Ù&þO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚663221932935680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95181595 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚746714548474368 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9717518161292 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166032722 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147945139 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52763542 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚10116.217655369 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166032724 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚672975353610752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95181597 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚9499 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166032722 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚663221950938624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211790848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166032724 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚736545103232000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚656020436295832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚672974562962944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚663221932935680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10169633093120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚656019648608060 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210554994 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 210554994 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚736545127172096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884980165764835e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147944608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52763013 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9753402672128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376610734 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚672975353610752 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9718288809704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚663221950938624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚746714760265216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210578010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95191243 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9718519301170 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚673043267648000 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376626301 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166049596 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210599747 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚551 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9754407431168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95191245 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 trainer/global_step‚9499 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376649345 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚663289650877952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10170484909568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166049598 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚553 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884980180474331e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376626301 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚10117.690854207 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376649345 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166049596 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147960111 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52768315 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9719289949790 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚663289650877952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166049598 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚736620475514880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52768866 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147959558 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚673044058309120 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚746790936484352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚663289632875008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚656087503871678 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9753634772992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10170683232768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚673043267648000 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376626301 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166049596 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210599747 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚551 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9754407431168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95191245 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 trainer/global_step‚9499 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376649345 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚663289650877952 TE8Ú:KDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10170484909568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166049598 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚553 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884980180474331e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376626301 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚10117.690854207 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376649345 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166049596 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147960111 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52768315 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9719289949790 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚663289650877952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166049598 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚736620475514880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52768866 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147959558 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚673044058309120 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚746790936484352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚663289632875008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚656087503871678 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9753634772992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10170683232768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95191243 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115‹†¾N4Ý&/ _wandbruntime‚10117  _runtime‚10117Ã}MÚ\Þ&Ö 2loss/SRqqt1/assignment_loss‚1.3893377078430993 1loss/FBt4/assignment_loss‚0.09185514676264735 /loss/FBt2/detection_loss‚2.5520380288362503 trainer/global_step‚9499 /loss/FRt1/detection_loss‚2.2251621431360644 1loss/SRqqt4/detection_loss‚0.8417875677026728 .loss/FBt1/detection_loss‚1.863622812288148 /loss/FBt4/detection_loss‚0.4551874922380145 _step‚780 0loss/FRt4/assignment_loss‚1.1650679646169437 2loss/SRqqt3/assignment_loss‚1.6441809458037218 .loss/FRt4/detection_loss‚1.454825453350649 %loss/total_loss‚39.59920973596883 % _timestamp‚1.7884980180518131e+09 2loss/SRqqt4/assignment_loss‚1.0478357506104774 1loss/SRqqt2/detection_loss‚2.4358518886566163 _runtime‚10117.694129597 0loss/FBt1/assignment_loss‚0.3425998618722567 1loss/FBt3/assignment_loss‚0.18195301989119095 .loss/FBt3/detection_loss‚2.224204202493032 /loss/FRt1/assignment_loss‚3.201376848750644 0loss/FRt2/assignment_loss‚2.7142853313876736 0loss/SRqqt1/detection_loss‚2.158187939652375 epoch‚7 0loss/FRt3/assignment_loss‚2.5285528942390725 2loss/SRqqt2/assignment_loss‚1.8243438374996186 0loss/FBt2/assignment_loss‚0.0847948490434381 .loss/FRt2/detection_loss‚2.587021651614568 /loss/FRt3/detection_loss‚2.7623515046305127 1loss/SRqqt3/detection_loss‚1.8227848930491342?*0˜aß&Û 1loss/SRqqt4/detection_loss‚0.8417875677026728 .loss/FBt1/detection_loss‚1.863622812288148 /loss/FBt4/detection_loss‚0.4551874922380145 _step‚780 0loss/FRt4/assignment_loss‚1.1650679646169437 2loss/SRqqt3/assignment_loss‚1.6441809458037218 .loss/FRt4/detection_loss‚1.454825453350649 %loss/total_loss‚39.59920973596883 % _timestamp‚1.7884980180518131e+09 2loss/SRqqt4/assignment_loss‚1.0478357506104774 1loss/SRqqt2/detection_loss‚2.4358518886566163 _runtime‚10117.694129597 0loss/FBt1/assignment_loss‚0.3425998618722567 1loss/FBt3/assignment_loss‚0.18195301989119095 .loss/FBt3/detection_loss‚2.224204202493032 /loss/FRt1/assignment_loss‚3.201376848750644 0loss/FRt2/assignment_loss‚2.7142853313876736 0loss/SRqqt1/detection_loss‚2.158187939652375 epoch‚7 0loss/FRt3/assignment_loss‚2.5285528942390725 2loss/SRqqt2/assignment_loss‚1.8243438374996186 0loss/FBt2/assignment_loss‚0.0847948490434381 .loss/FRt2/detection_loss‚2.587021651614568 /loss/FRt3/detection_loss‚2.7623515046305127 1loss/SRqqt3/detection_loss‚1.8227848930491342 2loss/SRqqt1/assignment_loss‚1.3893377078430993 1loss/FBt4/assignment_loss‚0.09185514676264735 /loss/FBt2/detection_loss‚2.5520380288362503 trainer/global_step‚9499 /loss/FRt1/detection_loss‚2.2251621431360644Œènà’à&‚(:‡ é éÔŸ±ð° gpu.0.gpu‚37.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2956 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä>* ‡á&‚(:ü é éÔ¤—Á³$ disk./.usageGB‚1172.600154876709" disk.nvme0n1.in‚31859.27734375 disk.sda3.out‚ 631.78515625 disk.sda.out‚ 631.78515625' disk.nvme0n1.out‚239464.93603515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 1118.71875# proc.memory.rssMB‚ 2949.64453125 cpu‚25.28515689718865 disk.dm-1.in‚0) disk./.usagePercent‚78.69418027083877 disk.dm-0.in‚31859.27734375 disk.sda3.in‚0* proc.memory.percent‚18.003201484680176$ disk.dm-0.out‚239464.93603515625 disk.dm-1.out‚ 631.78515625 disk.sda.in‚0$ disk.nvme0c0n1.in‚31859.27734375) disk.nvme0c0n1.out‚239464.93603515625$ memory_percent‚93.17188262939453 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.42578125 network.sent‚ 2.3982128e+07 network.recv‚ 3.329563e+06¢û‰¡’â&‚(:‡ ø éÔþ´œ¯ gpu.0.gpu‚33.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚31.0 gpu.0.powerWatts‚123.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02øݳ‚ã&‚(:÷ ø éÔšŽÝ¶$ disk.dm-0.out‚239665.38134765625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 32019.734375$ memory_percent‚93.17193031311035 disk.sda3.out‚ 632.6171875) disk.nvme0c0n1.out‚239665.38134765625( proc.memory.availableMB‚ 1118.7109375* proc.memory.percent‚18.003201484680176' disk.nvme0n1.out‚239665.38134765625 disk.drbd1260.out‚ 1.42578125 disk.sda.out‚ 632.6171875 disk.sda1.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2949.64453125 cpu‚25.285064936962762 proc.cpu.threads‚17 network.recv‚ 3.335522e+06) disk./.usagePercent‚78.69417694274082% disk./.usageGB‚1172.6001052856445 disk.dm-0.in‚ 32019.734375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 632.6171875 disk.sda1.out‚0 network.sent‚ 2.4092732e+07 disk.nvme0n1.in‚ 32019.7343751Ô^’ä&‚(:‡ ‡¡éÔþ’ú¬ gpu.0.gpu‚22.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°Ѐå&‚(:õ ‡¡éÔ¶ñË® network.sent‚ 2.409596e+07 disk.sda1.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1118.43359375* proc.memory.percent‚18.003201484680176 cpu‚25.284810337883332 proc.cpu.threads‚17 network.recv‚ 3.338924e+06) disk./.usagePercent‚78.71094159612055$ disk./.usageGB‚1172.849910736084 disk.drbd1260.in‚0 disk.sda.out‚ 633.03515625 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 32019.734375 disk.sda3.out‚ 633.03515625 disk.drbd1260.out‚ 1.4296875" disk.nvme0c0n1.in‚ 32019.734375& disk.nvme0n1.out‚239826.0537109375 disk.dm-0.in‚ 32019.734375# disk.dm-0.out‚239826.0537109375$ memory_percent‚93.17362308502197# proc.memory.rssMB‚ 2949.64453125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 633.03515625 disk.sda.in‚0( disk.nvme0c0n1.out‚239826.0537109375î+×Üæ&‚(:’ –¡éԦן® gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.226' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.245200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wê¬lç&‚(:á –¡éÔÈ£‚°# proc.memory.rssMB‚ 2949.64453125 cpu‚25.284581344804103 network.sent‚ 2.4103062e+07% disk./.usageGB‚1172.9827499389648 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚240041.3818359375* proc.memory.percent‚18.003201484680176 network.recv‚ 3.342508e+06 disk.dm-1.out‚633.5 disk.drbd1260.out‚ 1.4296875 disk.sda.out‚633.5 proc.cpu.threads‚17( disk./.usagePercent‚78.7198565464854 disk.dm-0.in‚ 32019.734375# disk.dm-0.out‚240041.3818359375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 32019.734375 disk.sda3.out‚633.5( disk.nvme0c0n1.out‚240041.3818359375 disk.nvme0n1.in‚ 32019.734375$ memory_percent‚93.17362308502197) proc.memory.availableMB‚ 1118.43359375¯ä 4è&/ _wandbruntime‚10175  _runtime‚10175Ö£íæÊé&Ä _step‚781 ' lr-AdamW/pg1‚0.00010190650521156607 ' lr-AdamW/pg2‚0.00010190650521156607 trainer/global_step‚9549 % _timestamp‚1.7884980754030044e+09 _runtime‚10175.0433586015êýyÏê&É ' lr-AdamW/pg1‚0.00010190650521156607 ' lr-AdamW/pg2‚0.00010190650521156607 trainer/global_step‚9549 % _timestamp‚1.7884980754030044e+09 _runtime‚10175.043358601 _step‚781u½Ö4ë&/ _wandbruntime‚10175  _runtime‚10175^S‚ƒÿ'ì&ùO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚659373828668940 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884980754040418e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9802881727488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 _runtime‚10175.044461309 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10221842340864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 378520318 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10222046968832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚676409811629568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 378543336 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚782 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166876424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211643896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚666606929902080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚666606947905024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚666606929902080 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211643896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚676409811629568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚740312544311296 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211666912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚649605452234768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9803654372352 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148692988 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166876424 trainer/global_step‚9549 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95663997 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚750534591280128 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚666606947905024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95663995 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9768359394812 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9803654372352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9802881727488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166876422 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204627968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211666912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚569 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53028424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚659373040981168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9803654372352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9802881727488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166876422 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204627968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211666912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚569 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53028424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚659373040981168 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10222046968832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚676409811629568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 378543336 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚782 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166876424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211643896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚666606929902080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚666606947905024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚666606929902080 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211643896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚676409811629568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚740312544311296 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211666912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚649605452234768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0Ž(ŒñÛ4î&/ _wandbruntime‚10175  _runtime‚10175e3™ 'ï&ƒN TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚666674629841408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9803886473216 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148707332 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚649671518670528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10222891218944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚676478516314624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166893296 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211688649 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚740387892654080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚666674647844352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95673645 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 378558903 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9803886473216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9769360534898 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378581947 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 378558903 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95673643 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚676479306975744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 211688649 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53033689 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211665607 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378581947 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚783 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚740387868713984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚659440896244786 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148707332 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚649671518670528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10222891218944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚676478516314624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166893296 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211688649 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚740387892654080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚666674647844352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95673645 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 378558903 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9803886473216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9769360534898 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378581947 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 378558903 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95673643 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚676479306975744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 211688649 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53033689 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211665607 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378581947 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚783 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚180029444Þl24ñ&/ _wandbruntime‚10175  _runtime‚10175l–^ò&Ø trainer/global_step‚9549 _runtime‚10175.904973992 0loss/SRqqt1/detection_loss‚2.557541982880954 1loss/SRqqt3/detection_loss‚1.9164658559616221 1loss/SRqqt4/detection_loss‚0.8003733100711691 /loss/FBt3/detection_loss‚2.2222353645733426 %loss/total_loss‚41.40847091959455 0loss/FRt1/assignment_loss‚3.0582398789269583 2loss/SRqqt4/assignment_loss‚1.1063952430826136 1loss/FBt4/assignment_loss‚0.08510230411848906 .loss/FRt1/detection_loss‚2.811891515978745 /loss/FRt3/detection_loss‚2.9248119875274856 1loss/SRqqt2/detection_loss‚2.4119314825212634 2loss/SRqqt2/assignment_loss‚1.3389833160751574 .loss/FRt4/detection_loss‚1.502120380892473 epoch‚7 /loss/FRt2/assignment_loss‚3.085192404295269 0loss/FRt3/assignment_loss‚2.0674072812343467 1loss/FBt3/assignment_loss‚0.34809376325990066 .loss/FBt1/detection_loss‚2.053033636166499 % _timestamp‚1.7884980762648199e+09 2loss/SRqqt3/assignment_loss‚1.6474448217208995 1loss/FBt1/assignment_loss‚0.28012404578303046 0loss/FBt2/assignment_loss‚0.2180838294847831 /loss/FBt2/detection_loss‚2.6475605964660645 /loss/FRt2/detection_loss‚2.6111400645476417 2loss/SRqqt1/assignment_loss‚1.6393056332037366 0loss/FRt4/assignment_loss‚1.4537080798078985 /loss/FBt4/detection_loss‚0.6212841410142107 _step‚7842Ũµcó&Ý trainer/global_step‚9549 _runtime‚10175.904973992 0loss/SRqqt1/detection_loss‚2.557541982880954 1loss/SRqqt3/detection_loss‚1.9164658559616221 1loss/SRqqt4/detection_loss‚0.8003733100711691 /loss/FBt3/detection_loss‚2.2222353645733426 %loss/total_loss‚41.40847091959455 0loss/FRt1/assignment_loss‚3.0582398789269583 2loss/SRqqt4/assignment_loss‚1.1063952430826136 1loss/FBt4/assignment_loss‚0.08510230411848906 .loss/FRt1/detection_loss‚2.811891515978745 /loss/FRt3/detection_loss‚2.9248119875274856 1loss/SRqqt2/detection_loss‚2.4119314825212634 2loss/SRqqt2/assignment_loss‚1.3389833160751574 .loss/FRt4/detection_loss‚1.502120380892473 epoch‚7 /loss/FRt2/assignment_loss‚3.085192404295269 0loss/FRt3/assignment_loss‚2.0674072812343467 1loss/FBt3/assignment_loss‚0.34809376325990066 .loss/FBt1/detection_loss‚2.053033636166499 % _timestamp‚1.7884980762648199e+09 2loss/SRqqt3/assignment_loss‚1.6474448217208995 1loss/FBt1/assignment_loss‚0.28012404578303046 0loss/FBt2/assignment_loss‚0.2180838294847831 /loss/FBt2/detection_loss‚2.6475605964660645 /loss/FRt2/detection_loss‚2.6111400645476417 2loss/SRqqt1/assignment_loss‚1.6393056332037366 0loss/FRt4/assignment_loss‚1.4537080798078985 /loss/FBt4/detection_loss‚0.6212841410142107 _step‚784é61@œô&‚(:‘ ¥¡éÔÑ犰 gpu.0.gpu‚9.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.454' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.890800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ/´1€õ&‚(:õ ¥¡éÔÀÿÁ°( disk./.usagePercent‚78.7282963468667$ disk.dm-0.out‚240394.17626953125 disk.dm-1.in‚0 disk.sda.out‚ 633.9140625 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 32019.734375' disk.nvme0n1.out‚240394.17626953125 disk.sda3.in‚0 disk.sda2.in‚0$ memory_percent‚93.17231178283691( proc.memory.availableMB‚ 1118.6484375 proc.cpu.threads‚17 disk.dm-1.out‚ 633.9140625 network.recv‚ 3.347283e+06% disk./.usageGB‚1173.1085090637207) disk.nvme0c0n1.out‚240394.17626953125 disk.dm-0.in‚ 32019.734375 cpu‚25.28433349944929" disk.nvme0c0n1.in‚ 32019.734375 disk.sda3.out‚ 633.9140625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.43359375 disk.sda.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2949.64453125* proc.memory.percent‚18.003201484680176 network.sent‚ 2.4120479e+07DYëÁ’ö&‚(:‡ ´¡éÔ–ݨ¯ gpu.0.gpu‚43.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.107' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y¡–±‡÷&‚(:ü ´¡éÔ¨ž°²* proc.memory.percent‚18.003201484680176 cpu‚25.284030785407737 proc.cpu.threads‚17 network.recv‚ 3.35334e+06) disk./.usagePercent‚78.75352716940375 disk.drbd1260.out‚ 1.43359375 disk.sda.in‚0& disk.nvme0n1.out‚240745.9619140625 disk.dm-0.in‚32143.94140625# memory_percent‚93.1753396987915# proc.memory.rssMB‚ 2949.64453125% disk./.usageGB‚1173.4844665527344$ disk.nvme0c0n1.in‚32143.94140625# disk.dm-0.out‚240745.9619140625) proc.memory.availableMB‚ 1118.15234375 network.sent‚ 2.4216466e+07 disk.sda3.in‚0 disk.sda3.out‚ 634.26171875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 634.26171875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 634.26171875 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚240745.9619140625" disk.nvme0n1.in‚32143.94140625¦ÄmÑø&‚(:’ áéÔºî° gpu.0.gpu‚41.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚145.686' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.137200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÈ2ìxù&‚(:í áéÔø¥ü² proc.cpu.threads‚17 network.sent‚ 2.4221304e+07 disk.dm-1.in‚0$ disk.nvme0c0n1.out‚ 241195.546875" disk.nvme0n1.in‚32143.94140625 disk.dm-0.in‚32143.94140625 network.recv‚ 3.356469e+06) disk./.usagePercent‚78.78496771069133$ disk./.usageGB‚1173.952953338623 disk.sda3.out‚ 634.88671875 disk.drbd1260.in‚0 disk.sda.out‚ 634.88671875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 634.88671875 disk.drbd1260.out‚1.4375$ disk.nvme0c0n1.in‚32143.94140625" disk.nvme0n1.out‚ 241195.546875 disk.dm-0.out‚ 241195.546875$ memory_percent‚93.17352771759033# proc.memory.rssMB‚ 2949.64453125 cpu‚25.283783883448404 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1118.44921875* proc.memory.percent‚18.003201484680176üËA4ú&/ _wandbruntime‚10225  _runtime‚10225D(¢EÇû&Á & lr-AdamW/pg1‚9.810832343698134e-05 & lr-AdamW/pg2‚9.810832343698134e-05 trainer/global_step‚9599 $ _timestamp‚1.788498125732322e+09 _runtime‚10225.372703748 _step‚785ÌxðsÌü&Æ trainer/global_step‚9599 $ _timestamp‚1.788498125732322e+09 _runtime‚10225.372703748 _step‚785 & lr-AdamW/pg1‚9.810832343698134e-05 & lr-AdamW/pg2‚9.810832343698134e-05‘Æj 4ý&/ _wandbruntime‚10225  _runtime‚10225Þä´¯÷þ&ùO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212732798 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚754354186562560 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10274249052160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 trainer/global_step‚9599 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212755814 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149441602 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96146395 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚679845060296192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96146397 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚662726433354276 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212755814 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚679845850944000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167720124 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167720124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚679845850944000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚669991944871424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167720122 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚662727221042048 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167720122 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚652908774022768 % _timestamp‚1.7884981257334335e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53295207 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9817659331508 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 _runtime‚10225.373863657 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380475938 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚679845060296192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚744079961450496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚786 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10274447388672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944:8*`(ÿ&þO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚662726433354276 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212755814 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚679845850944000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167720124 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167720124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚679845850944000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚669991944871424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167720122 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚662727221042048 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167720122 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚652908774022768 % _timestamp‚1.7884981257334335e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53295207 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9817659331508 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 _runtime‚10225.373863657 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380475938 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚679845060296192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚744079961450496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚786 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10274447388672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212732798 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚754354186562560 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10274249052160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 trainer/global_step‚9599 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212755814 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149441602 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96146395 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚679845060296192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96146397 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512’fåµ4€'/ _wandbruntime‚10226  _runtime‚10226!¶êü ''ƒN [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚670059644810752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167736996 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10275297240064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167736998 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚662793500929914 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 380491505 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53300409 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212754509 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚662794288617894 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212754509 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚652974857497888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚652974840458528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 % _timestamp‚1.7884981265849526e+09 _step‚787 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚670059626807808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212777551 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96156043 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9818660471386 _runtime‚10226.226400366 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167736998 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167736996 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9854910831616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9854910831616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9854138173440 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚670059644810752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚744155285853184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53300934 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚754430803259392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚744155309793280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380514549 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚679914555642368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149456979 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149456452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328+Ýøí” ‚'ˆN I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚670059626807808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212777551 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96156043 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9818660471386 _runtime‚10226.226400366 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167736998 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167736996 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9854910831616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9854910831616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9854138173440 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 101967462©m5z4 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚670059644810752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚744155285853184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53300934 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚754430803259392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚744155309793280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380514549 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚679914555642368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149456979 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149456452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚670059644810752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167736996 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10275297240064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167736998 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚662793500929914 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 380491505 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53300409 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212754509 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚662794288617894 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212754509 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚652974857497888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚652974840458528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 % _timestamp‚1.7884981265849526e+09 _step‚787“ût×04ƒ'/ _wandbruntime‚10226  _runtime‚10226&ÖY„'Ó % _timestamp‚1.7884981265879128e+09 2loss/SRqqt2/assignment_loss‚1.4660217602705132 1loss/FBt3/assignment_loss‚0.17713308903486474 1loss/FBt4/assignment_loss‚0.13214391395508213 /loss/FRt4/detection_loss‚1.4649108294087152 0loss/SRqqt1/detection_loss‚2.455869972705841 /loss/FBt1/detection_loss‚2.0403233488400776 %loss/total_loss‚39.10617069515118 1loss/SRqqt1/assignment_loss‚1.505034180978934 1loss/FBt1/assignment_loss‚0.26879948272830867 /loss/FBt4/detection_loss‚0.5497762695502709 0loss/FRt4/assignment_loss‚0.9311854471762975 .loss/FBt2/detection_loss‚3.061725527048111 _step‚788 1loss/FBt2/assignment_loss‚0.07805014139603372 /loss/SRqqt3/detection_loss‚1.76565586602272 epoch‚7 .loss/FRt2/detection_loss‚2.572352320554745 .loss/FBt3/detection_loss‚2.323049936029646 0loss/FRt3/assignment_loss‚2.2973411021561456 2loss/SRqqt3/assignment_loss‚1.5998218499643857 2loss/SRqqt4/assignment_loss‚0.8772139762149891 0loss/SRqqt2/detection_loss‚2.301848214248131 0loss/FRt2/assignment_loss‚2.2706811405994274 .loss/FRt3/detection_loss‚2.970674932773771 trainer/global_step‚9599 _runtime‚10226.228080567 /loss/FRt1/assignment_loss‚2.748711273405287 .loss/FRt1/detection_loss‚2.535353721015983 1loss/SRqqt4/detection_loss‚0.7124923990728954D'{0^…'Ø 0loss/FRt4/assignment_loss‚0.9311854471762975 .loss/FBt2/detection_loss‚3.061725527048111 _step‚788 1loss/FBt2/assignment_loss‚0.07805014139603372 /loss/SRqqt3/detection_loss‚1.76565586602272 epoch‚7 .loss/FRt2/detection_loss‚2.572352320554745 .loss/FBt3/detection_loss‚2.323049936029646 0loss/FRt3/assignment_loss‚2.2973411021561456 2loss/SRqqt3/assignment_loss‚1.5998218499643857 2loss/SRqqt4/assignment_loss‚0.8772139762149891 0loss/SRqqt2/detection_loss‚2.301848214248131 0loss/FRt2/assignment_loss‚2.2706811405994274 .loss/FRt3/detection_loss‚2.970674932773771 trainer/global_step‚9599 _runtime‚10226.228080567 /loss/FRt1/assignment_loss‚2.748711273405287 .loss/FRt1/detection_loss‚2.535353721015983 1loss/SRqqt4/detection_loss‚0.7124923990728954 % _timestamp‚1.7884981265879128e+09 2loss/SRqqt2/assignment_loss‚1.4660217602705132 1loss/FBt3/assignment_loss‚0.17713308903486474 1loss/FBt4/assignment_loss‚0.13214391395508213 /loss/FRt4/detection_loss‚1.4649108294087152 0loss/SRqqt1/detection_loss‚2.455869972705841 /loss/FBt1/detection_loss‚2.0403233488400776 %loss/total_loss‚39.10617069515118 1loss/SRqqt1/assignment_loss‚1.505034180978934 1loss/FBt1/assignment_loss‚0.26879948272830867 /loss/FBt4/detection_loss‚0.5497762695502709”{Òå’†'‚(:‡ Ò¡éÔ›¬¿® gpu.0.gpu‚20.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.057' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ÆX„‡'‚(:ù Ò¡éÔ®ù¥± disk.sda1.out‚0) disk./.usagePercent‚78.79335298146783% disk./.usageGB‚1174.0778999328613 disk.sda3.out‚ 635.26171875 disk.dm-1.in‚0 disk.dm-1.out‚ 635.26171875 disk.drbd1260.out‚1.4375 disk.sda2.out‚0 network.recv‚ 3.359747e+06 disk.sda3.in‚0 disk.sda.in‚0$ disk.nvme0c0n1.in‚32143.94140625" disk.nvme0n1.in‚32143.94140625& disk.nvme0n1.out‚241500.2685546875# disk.dm-0.out‚241500.2685546875$ memory_percent‚93.17355155944824 disk.sda.out‚ 635.26171875 disk.sda1.in‚0( disk.nvme0c0n1.out‚241500.2685546875 disk.dm-0.in‚32143.94140625( proc.memory.availableMB‚ 1118.4453125# proc.memory.rssMB‚ 2949.64453125 cpu‚25.283524997886555 proc.cpu.threads‚17 network.sent‚ 2.4225584e+07 disk.drbd1260.in‚0 disk.sda2.in‚0* proc.memory.percent‚18.003201484680176¼km ‘ˆ'‚(:† á¡éÔ´ú™® gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚139.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.9386 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CPL‰'‚(:ö á¡éÔýôܲ$ disk./.usageGB‚1174.704818725586 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 635.69921875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 635.69921875 disk.drbd1260.out‚ 1.4453125 disk.sda.out‚ 635.69921875 disk.sda1.out‚0' disk.nvme0c0n1.out‚241805.380859375" disk.nvme0n1.in‚32143.94140625 proc.cpu.threads‚17 network.recv‚ 3.366251e+06 disk.drbd1260.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚32143.94140625 disk.dm-0.in‚32143.94140625' proc.memory.availableMB‚ 1118.640625# proc.memory.rssMB‚ 2949.64453125* proc.memory.percent‚18.003201484680176( disk./.usagePercent‚78.8354260276675 disk.sda2.out‚0% disk.nvme0n1.out‚241805.380859375" disk.dm-0.out‚241805.380859375$ memory_percent‚93.17235946655273 cpu‚25.283299554854224 network.sent‚ 2.4336697e+07-rª}’Š'‚(:‡ ð¡éÔ•ó­ gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚130.548' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.1096 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|þ$ò€‹'‚(:õ ð¡éÔžÚ¬³ disk.dm-1.in‚0# proc.memory.rssMB‚ 2949.64453125 network.sent‚ 2.4340741e+07 disk.sda.out‚ 636.1171875 disk.sda1.in‚0' disk.nvme0c0n1.out‚242204.818359375 disk.sda3.out‚ 636.1171875 disk.dm-1.out‚ 636.1171875 disk.drbd1260.in‚0' proc.memory.availableMB‚ 1118.390625 network.recv‚ 3.370657e+06$ disk.nvme0c0n1.in‚32282.69921875% disk.nvme0n1.out‚242204.818359375" disk.dm-0.out‚242204.818359375 disk.drbd1260.out‚ 1.4453125 proc.cpu.threads‚17% disk./.usageGB‚1174.9538497924805 disk.sda.in‚0 disk.dm-0.in‚32282.69921875 disk.sda3.in‚0$ memory_percent‚93.17388534545898* proc.memory.percent‚18.003201484680176 cpu‚25.283030647033385) disk./.usagePercent‚78.85213871151777 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚32282.69921875²ÕþYŒ'‚(:’ ÿ¡éÔ¨Õê® gpu.0.gpu‚14.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.129599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó!{…'‚(:ú ÿ¡éÔ„ð˱ network.sent‚ 2.4347911e+07 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚32288.61328125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0) disk./.usagePercent‚78.85206037321228% disk./.usageGB‚1174.9526824951172 disk.sda3.out‚ 636.421875 disk.sda.out‚ 636.421875) proc.memory.availableMB‚ 1118.88671875# proc.memory.rssMB‚ 2949.64453125$ disk.nvme0c0n1.in‚32288.61328125) disk.nvme0c0n1.out‚242601.95751953125$ disk.dm-0.out‚242601.95751953125 disk.dm-1.out‚ 636.421875# memory_percent‚93.1708574295044* proc.memory.percent‚18.003201484680176 cpu‚25.282822474266254 proc.cpu.threads‚17 network.recv‚ 3.374253e+06 disk.sda1.in‚0' disk.nvme0n1.out‚242601.95751953125 disk.dm-0.in‚32288.61328125 disk.drbd1260.out‚ 1.44921875 disk.sda.in‚0§[W4Ž'/ _wandbruntime‚10275  _runtime‚10275 å}[È' & lr-AdamW/pg2‚9.437224706880018e-05 trainer/global_step‚9649 % _timestamp‚1.7884981761423583e+09 _runtime‚10275.782788733 _step‚789 & lr-AdamW/pg1‚9.437224706880018e-05´8¥•Í'Ç _step‚789 & lr-AdamW/pg1‚9.437224706880018e-05 & lr-AdamW/pg2‚9.437224706880018e-05 trainer/global_step‚9649 % _timestamp‚1.7884981761423583e+09 _runtime‚10275.782788733•»Dƾ4‘'/ _wandbruntime‚10275  _runtime‚10275ȵâÿ'’'ùO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚673376941837824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚666079825727384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9867729916616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213821700 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96628797 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚673376941837824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9903385127936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚683280308962816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚683280308962816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9904157772800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _step‚790 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213821700 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53561780 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150190575 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚747847354649600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚10275.784083074 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚673376923834880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚758174085551616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 % _timestamp‚1.7884981761435418e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656212095810768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9868500565028 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚683281099610624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53562325 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚683281099610624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 382385522 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168563824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213844716 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10326730902016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9904157772800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10326925044224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150191122 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚666080613415156 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚9649 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 213844716 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352‘"õðÅ“'þO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213821700 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53561780 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150190575 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚747847354649600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚10275.784083074 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚673376923834880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚758174085551616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 % _timestamp‚1.7884981761435418e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656212095810768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9868500565028 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚683281099610624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53562325 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚683281099610624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 382385522 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168563824 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213844716 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10326730902016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9904157772800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10326925044224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150191122 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚666080613415156 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 trainer/global_step‚9649 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 213844716 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚673376941837824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚666079825727384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9867729916616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213821700 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96628797 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚673376941837824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9903385127936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚683280308962816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚683280308962816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9904157772800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _step‚790–·+à4”'/ _wandbruntime‚10277  _runtime‚10277/Æ‘ '•'ƒN \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚758250700160512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150205550 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚673444641777152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213843411 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382424107 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9905162531840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213866453 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53567640 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96638443 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9649 _step‚791 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚666146893303022 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚10277.105483495 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚747922702992384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168580698 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168580696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9904389873664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213843411 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9905162531840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚666147680991002 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10327777001984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382447151 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168580696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382424107 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9905162531840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213866453 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53567640 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96638443 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9649 _step‚791 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚666146893303022 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚10277.105483495 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚747922702992384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168580698 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168580696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9904389873664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213843411 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9905162531840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚666147680991002 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10327777001984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382447151 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168580696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150205550 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚673444641777152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213843411 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 F„£'‚(:ù Ê¢éÔ‘ر³ disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚93.17028522491455 disk.sda3.out‚ 638.859375 disk.sda.out‚ 638.859375$ disk.nvme0c0n1.in‚32450.87890625) disk.nvme0c0n1.out‚244399.64501953125" disk.nvme0n1.in‚32450.87890625' disk.nvme0n1.out‚244399.64501953125* proc.memory.percent‚18.003201484680176 proc.cpu.threads‚17 network.sent‚ 2.4478152e+07 network.recv‚ 3.395933e+06$ disk.dm-0.out‚244399.64501953125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.45703125 disk.sda1.in‚0) proc.memory.availableMB‚ 1118.98046875) disk./.usagePercent‚78.88562628904914 disk.dm-0.in‚32450.87890625 disk.sda3.in‚0 disk.dm-1.out‚ 638.859375 disk.sda2.out‚0# proc.memory.rssMB‚ 2949.64453125 cpu‚25.30414934458416$ disk./.usageGB‚1175.452838897705 disk.sda.in‚0ì—l4¤'/ _wandbruntime‚10359  _runtime‚10359 Ä…X¥'àT LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28115 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚539364192PN) \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 % _timestamp‚1.7884982595963843e+09 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27846 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 10929229312 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18217 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚687111432315904 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚762256298535424 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚10373978942976 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1005 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1005 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18217 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 9949477914112 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 169587552 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 375175680 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 151178138 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 580044288 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 214750613 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27846 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882825 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 384319950 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 214750613 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 214732398 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 _step‚793 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚677161972404736 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 152733184 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578072504 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚687112030363136 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚677161972404736 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚97241721 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 169587554 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 12155092992 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚677161954401792 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 384338167 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18215 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚97241719 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚762256145802240 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111864 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 598047232 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚751882319592448 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 128793088 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚39 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 384338167 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 169587552 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18215 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚677161954401792 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 9950057958400 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚10373850149888 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3125 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚53936604 trainer/global_step‚9671±4åWk*¦'åT WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚687111432315904 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚187 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 169587554 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚97241719 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚762256145802240 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111864 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 598047232 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚751882319592448 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 128793088 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚39 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 384338167 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 169587552 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18215 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚677161954401792 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 9950057958400 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚10373850149888 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3125 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚53936604 trainer/global_step‚9671 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28115 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚53936419 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 % _timestamp‚1.7884982595963843e+09 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27846 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 10929229312 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18217 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚687111432315904 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚762256298535424 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚10373978942976 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1005 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1005 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18217 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 9949477914112 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 169587552 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 375175680 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 151178138 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 580044288 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 214750613 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27846 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882825 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 384319950 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 214750613 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 214732398 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 _step‚793 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚677161972404736 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 152733184 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578072504 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚687112030363136 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚677161972404736 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚97241721 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 169587554 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 12155092992 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚677161954401792 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 384338167 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18215 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12155092992 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 _runtime‚10360.35297599 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 580044288 % _timestamp‚1.7884982607121723e+09 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595111864 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 598047232 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚751929273797632 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 214758907 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 151191220 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚10374443691520 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12155092992 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3125 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚687157365063680 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 128793088 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892004352 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 384359433 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚669761137251906 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚762303588696064 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9914704374498 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 169600526 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚687157963110912 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚687157963110912 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 375175680 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 578072504 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2™ýÇÁ©'îR WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 595111864 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 598047232 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚751929273797632 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚489 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 214758907 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 151jPój³&191220 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚10374443691520 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 12155092992 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3125 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚687157365063680 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 128793088 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892004352 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 384359433 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚669761137251906 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚489 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚762303588696064 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9914704374498 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 169600526 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚687157963110912 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚687157963110912 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 375175680 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 578072504 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚489 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 _runtime‚10360.35297599 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 580044288 % _timestamp‚1.7884982607121723e+09šJIR@ª'‚(:‚ Ù¢éÔ¢¥œ° gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.4' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.68 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½èÈÞ‚«'‚(:÷ Ù¢éÔþ»² network.recv‚ 3.400782e+06) disk./.usagePercent‚78.88562628904914$ disk./.usageGB‚1175.452838897705 disk.dm-0.in‚32450.87890625 disk.sda3.in‚0 disk.sda.out‚ 639.640625 disk.sda1.out‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚245016.66357421875' disk.nvme0n1.out‚245016.66357421875 disk.dm-1.in‚0 disk.dm-1.out‚ 639.640625 disk.drbd1260.out‚ 1.4609375 disk.sda.in‚0 disk.sda2.in‚0# memory_percent‚93.1720495223999 network.sent‚ 2.449564e+07" disk.nvme0n1.in‚32450.87890625$ disk.dm-0.out‚245016.66357421875 disk.sda3.out‚ 639.640625 disk.drbd1260.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2949.64453125* proc.memory.percent‚18.003201484680176$ disk.nvme0c0n1.in‚32450.87890625) proc.memory.availableMB‚ 1118.69140625 cpu‚25.310635873344918 proc.cpu.threads‚17Õt©4¬'/ _wandbruntime‚10375  _runtime‚10375 ò:ÄçS­'ܦ ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11259136348962784 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5494349598884583 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5121862292289734 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5133496522903442 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.8125 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.05711085721850395 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12193524092435837 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3500902056694031 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123459696769714 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5518674254417419 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9841400980949402 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5604252219200134 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6399157643318176 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.193452388048172 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_6_of_7‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13039088249206543 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.984695553779602 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1982758492231369 particle/accuracy_1_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.014874551445245743 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4305555522441864 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.0855855792760849 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30795639753341675 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5650103092193604 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29197800159454346 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666865348816 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19110770523548126 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.705451250076294 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4957958459854126 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23060007393360138 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.16858239471912384 particle/accuracy_1_of_3‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,particle/sensitivity‚0.34669405221939087 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42444220185279846 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3136596381664276 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6145833134651184 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290780425071716 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5044541954994202 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22155947983264923 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04992246255278587 particle/accuracy_7_of_7‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33482226729393005 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5981229543685913 *particle/specificity‚0.947977602481842 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9093567728996277 particle/accuracy_6_of_6‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9712203145027161 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.565874457359314 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 jet/accuracy_2_of_7‚0.5 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14127634465694427 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4920351803302765 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10464473068714142 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9798850417137146 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1563786119222641 +jet/accuracy_3_of_5‚0.09992318600416183 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992063641548157 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5685344338417053 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22345955669879913 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7424241900444031 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4756944477558136 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3635711371898651 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002543752547353506 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007862962782382965 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978000521659851 particle/accuracy_2_of_4‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6805555820465088 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5645018815994263 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5544015169143677 jet/accuracy_5_of_6‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 &particle/f_score‚0.425735205411911 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9912068247795105 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.989861011505127 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5432128310203552 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3907386064529419 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38402774930000305 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.005423280410468578 *jet/accuracy_2_of_3‚0.2761732339859009 particle/accuracy_1_of_5‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 particle/accuracy_2_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 *"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9780722856521606 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5203782320022583 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5269179344177246 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6479938626289368 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3125174939632416 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36481472849845886 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981828927993774 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8540735840797424 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5159346461296082 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02840193547308445 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 jet/accuracy_4_of_7‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.13333334028720856 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30517247319221497 *jet/accuracy_1_of_4‚0.9284588694572449 jet/accuracy_7_of_7‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4974697232246399 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04326145723462105 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9981560111045837 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4068782925605774 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9678819179534912 particle/accuracy_4_of_5‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.594924807548523 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9947922229766846 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.6050843596458435 *jet/accuracy_1_of_5‚0.9643881320953369 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17717067897319794 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08947920054197311 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5111227035522461 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.6875 jet/accuracy_5_of_5‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011968175880610943 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.337797611951828 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_5_of_6‚1 particle/accuracy_4_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23038320243358612 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.08230172097682953 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5121862292289734 +jet/accuracy_3_of_3‚0.02926267497241497 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4859794080257416 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42280733585357666 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.36827558279037476 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.027419503778219223 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 % _timestamp‚1.7884982761491628e+09 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5700286626815796 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5329456925392151 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9957482814788818 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992063641548157 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6152777075767517 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23528315126895905 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10638384521007538 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5524151921272278 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11260806769132614 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6003085970878601 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27475208044052124 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5550763607025146 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.013888888992369175 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.10952940583229065 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9898033738136292 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.54666668176651 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16500765085220337 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4062499701976776 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11773616820573807 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7683186531066895 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9904884099960327 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.0983319953083992 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5950356125831604 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22155947983264923 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4305555522441864 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24011869728565216 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 *"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.5 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 )loss/val_total_loss‚40.02502334305154 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26578420400619507 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.10519292950630188 *jet/accuracy_1_of_1‚0.4985979199409485 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20743921399116516 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 epoch‚7 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9881154894828796 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2076593041419983 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1840638518333435 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12039241194725037 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5421798229217529 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5756579041481018 ,&Purity/2FRt1SRqqt3FBt/evôÞp|?ent_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13234446942806244 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5606228709220886 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7575757503509521 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.45958274602890015 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 _runtime‚10375.791406182 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18988487124443054 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9888925552368164 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.041173242032527924 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5345984101295471 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2583567500114441 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5796325206756592 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.48442259430885315 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9897148609161377 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.08187135308980942 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.650011420249939 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19038696587085724 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2260802537202835 jet/accuracy_1_of_6‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.41798827052116394 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03943449258804321 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14832694828510284 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15555556118488312 _step‚795 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.10775861889123917 particle/accuracy_1_of_7‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0010582010727375746 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5814751982688904 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5462969541549683 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_6‚0.49358975887298584 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.929894208908081 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5933907628059387 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 *jet/accuracy_2_of_5‚0.5194187760353088 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7456140518188477 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5459017753601074 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5674816966056824 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9837301969528198 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03191600367426872 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.41134220361709595 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0440068319439888 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 particle/accuracy_4_of_6‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8158866763114929 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6387820839881897 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.12182294577360153 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5865474939346313 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9893198609352112 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1814165860414505 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.31296297907829285 +jet/accuracy_3_of_6‚0.13461539149284363 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996031522750854 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.45067140460014343 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6666666865348816 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23667997121810913 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09568235278129578 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4512680470943451 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851478934288025 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.615879476070404 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2316809743642807 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.2348484843969345 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20396865904331207 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17719046771526337 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5832437872886658 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6141295433044434 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.022373395040631294 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_3_of_7‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 particle/accuracy_1_of_4‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5697821974754333 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 trainer/global_step‚9671 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20077750086784363 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.015517241321504116 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5088962912559509 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.481321781873703 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 particle/accuracy_5_of_5‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5333333611488342 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19438260793685913 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5774886012077332 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4092261791229248 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5095959901809692 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17445223033428192 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.6666667461395264 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4876905679702759 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.15625 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6934523582458496 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6666666865348816 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.016364049166440964 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.028931036591529846 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9236111640930176 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5352872014045715 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9880219101905823 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.237615168094635 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6110639572143555 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_3‚0.8915802240371704 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3133307099342346 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07247023284435272 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24436156451702118 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2828071415424347 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666865348816 jet/accuracy_4_of_6‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚1 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.590783417224884 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.48106062412261963 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.08459595590829849 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968417882919312 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4871344268321991 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5511988997459412 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.991421103477478 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6764705777168274 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.10682187229394913 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0:‹:mv@®'ᦠ:"Purity/1FRt*SRqqt1FBt/event_purity‚0.17717067897319794 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08947920054197311 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5111227035522461 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.6875 jet/accuracy_5_of_5‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011968175880610943 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.337797611951828 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_5_of_6‚1 particle/accuracy_4_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23038320243358612 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.08230172097682953 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5121862292289734 +jet/accuracy_3_of_3‚0.02926267497241497 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4859794080257416 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42280733585357666 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.36827558279037476 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.027419503778219223 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 % _timestamp‚1.7884982761491628e+09 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5700286626815796 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5329456925392151 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9957482814788818 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992063641548157 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6152777075767517 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23528315126895905 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10638384521007538 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5524151921272278 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11260806769132614 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6003085970878601 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27475208044052124 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5550763607025146 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.013888888992369175 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.10952940583229065 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9898033738136292 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.54666668176651 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16500765085220337 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4062499701976776 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.019607843831181526 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11773616820573807 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7683186531066895 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9904884099960327 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.0983319953083992 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5950356125831604 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22155947983264923 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4305555522441864 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24011869728565216 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 *"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.5 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 )loss/val_total_loss‚40.02502334305154 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26578420400619507 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.10519292950630188 *jet/accuracy_1_of_1‚0.4985979199409485 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20743921399116516 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 epoch‚7 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9881154894828796 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2076593041419983 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1840638518333435 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12039241194725037 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5421798229217529 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5756579041481018 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13234446942806244 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5606228709220886 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7575757503509521 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.45958274602890015 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.019607843831181526 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 _runtime‚10375.791406182 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18988487124443054 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9888925552368164 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.041173242032527924 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5345984101295471 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2583567500114441 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5796325206756592 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.48442259430885315 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9897148609161377 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.08187135308980942 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.650011420249939 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19038696587085724 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2260802537202835 jet/accuracy_1_of_6‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.41798827052116394 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03943449258804321 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14832694828510284 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15555556118488312 _step‚795 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.10775861889123917 particle/accuracy_1_of_7‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0010582010727375746 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5814751982688904 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5462969541549683 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_6‚0.49358975887298584 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.929894208908081 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5933907628059387 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 *jet/accuracy_2_of_5‚0.5194187760353088 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7456140518188477 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5459017753601074 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5674816966056824 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9837301969528198 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03191600367426872 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.41134220361709595 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0440068319439888 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 particle/accuracy_4_of_6‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8158866763114929 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6387820839881897 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.12182294577360153 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5865474939346313 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9893198609352112 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1814165860414505 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.31296297907829285 +jet/accuracy_3_of_6‚0.13461539149284363 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996031522750854 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.45067140460014343 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6666666865348816 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23667997121810913 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09568235278129578 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4512680470943451 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851478934288025 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.615879476070404 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2316809743642807 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.2348484843969345 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20396865904331207 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17719046771526337 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5832437872886658 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6141295433044434 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.022373395040631294 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_3_of_7‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 particle/accuracy_1_of_4‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5697821974754333 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 trainer/global_step‚9671 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20077750086784363 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.015517241321504116 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBÁì“1òRt/FBt_purity‚0.875 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5088962912559509 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.481321781873703 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 particle/accuracy_5_of_5‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5333333611488342 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19438260793685913 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5774886012077332 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4092261791229248 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5095959901809692 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17445223033428192 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.6666667461395264 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4876905679702759 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.15625 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6934523582458496 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6666666865348816 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.016364049166440964 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.028931036591529846 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9236111640930176 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5352872014045715 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9880219101905823 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.237615168094635 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6110639572143555 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_3‚0.8915802240371704 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3133307099342346 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07247023284435272 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24436156451702118 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2828071415424347 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666865348816 jet/accuracy_4_of_6‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚1 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.590783417224884 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.48106062412261963 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.08459595590829849 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968417882919312 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4871344268321991 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5511988997459412 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.991421103477478 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6764705777168274 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.10682187229394913 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11259136348962784 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5494349598884583 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5121862292289734 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5133496522903442 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.8125 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.05711085721850395 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12193524092435837 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3500902056694031 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123459696769714 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5518674254417419 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9841400980949402 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5604252219200134 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6399157643318176 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.193452388048172 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_6_of_7‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13039088249206543 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.984695553779602 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1982758492231369 particle/accuracy_1_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.014874551445245743 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4305555522441864 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.0855855792760849 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30795639753341675 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5650103092193604 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29197800159454346 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666865348816 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19110770523548126 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.705451250076294 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4957958459854126 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.23060007393360138 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.16858239471912384 particle/accuracy_1_of_3‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,particle/sensitivity‚0.34669405221939087 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42444220185279846 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3136596381664276 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6145833134651184 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290780425071716 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5044541954994202 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22155947983264923 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04992246255278587 particle/accuracy_7_of_7‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33482226729393005 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5981229543685913 *particle/specificity‚0.947977602481842 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9093567728996277 particle/accuracy_6_of_6‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9712203145027161 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.565874457359314 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 jet/accuracy_2_of_7‚0.5 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14127634465694427 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4920351803302765 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10464473068714142 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9798850417137146 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1563786119222641 +jet/accuracy_3_of_5‚0.09992318600416183 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992063641548157 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5685344338417053 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22345955669879913 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7424241900444031 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4756944477558136 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3635711371898651 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002543752547353506 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007862962782382965 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978000521659851 particle/accuracy_2_of_4‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6805555820465088 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5645018815994263 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5544015169143677 jet/accuracy_5_of_6‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 &particle/f_score‚0.425735205411911 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9912068247795105 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.989861011505127 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5432128310203552 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3907386064529419 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38402774930000305 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.005423280410468578 *jet/accuracy_2_of_3‚0.2761732339859009 particle/accuracy_1_of_5‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 particle/accuracy_2_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 *"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9780722856521606 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5203782320022583 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5269179344177246 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6479938626289368 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3125174939632416 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36481472849845886 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981828927993774 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8540735840797424 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5159346461296082 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02840193547308445 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 jet/accuracy_4_of_7‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.13333334028720856 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30517247319221497 *jet/accuracy_1_of_4‚0.9284588694572449 jet/accuracy_7_of_7‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4974697232246399 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04326145723462105 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9981560111045837 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4068782925605774 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9678819179534912 particle/accuracy_4_of_5‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.594924807548523 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9947922229766846 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.6050843596458435 *jet/accuracy_1_of_5‚0.9643881320953369 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5›ªézß›¯'‚(: è¢éÔœÈç¬ gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.564' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.712799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Û*…°'‚(:ú è¢éÔ’ο¯$ disk.nvme0c0n1.in‚32450.87890625' disk.nvme0n1.out‚245272.84326171875 disk.sda3.out‚ 640.0078125) disk./.usagePercent‚78.88578296566008% disk./.usageGB‚1175.4551734924316$ disk.dm-0.out‚245272.84326171875 disk.dm-1.in‚0) proc.memory.availableMB‚ 1532.33203125" proc.memory.rssMB‚ 3096.2421875 network.recv‚ 3.40726e+06 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚90.64738750457764) proc.memory.percent‚18.89796257019043 cpu‚25.315342214324566 proc.cpu.threads‚17 network.sent‚ 2.4591371e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚245272.84326171875" disk.nvme0n1.in‚32450.87890625 disk.dm-0.in‚32450.87890625 disk.dm-1.out‚ 640.0078125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.4609375 disk.sda.in‚0 disk.sda.out‚ 640.00781255þŸí‘±'‚(:† ÷¢éÔ™Ú‰° gpu.0.gpu‚32.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚158.715' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.743 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âbh‡²'‚(:ü ÷¢éԼϳ network.recv‚ 3.412302e+06 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 640.30078125 disk.sda2.out‚0 disk.dm-0.in‚32585.60546875# disk.dm-0.out‚246715.5693359375 disk.dm-1.in‚0 disk.dm-1.out‚ 640.30078125 disk.sda2.in‚0$ disk.nvme0c0n1.in‚32585.60546875( disk.nvme0c0n1.out‚246715.5693359375" disk.nvme0n1.in‚32585.60546875) proc.memory.availableMB‚ 1678.59765625 network.sent‚ 2.471467e+07% disk./.usageGB‚1175.4529609680176 disk.drbd1260.out‚ 1.46484375 disk.sda1.in‚0* proc.memory.percent‚18.003201484680176 cpu‚25.31451209026329) disk./.usagePercent‚78.88563448129024 disk.sda3.in‚0 disk.sda3.out‚ 640.30078125 disk.sda1.out‚0& disk.nvme0n1.out‚246715.5693359375$ memory_percent‚89.75465297698975# proc.memory.rssMB‚ 2949.64453125 proc.cpu.threads‚17ƒA@'4³'/ _wandbruntime‚10409  _runtime‚104093A˜pÈ´' _runtime‚10409.890627925 _step‚796 & lr-AdamW/pg1‚9.069905469973783e-05 & lr-AdamW/pg2‚9.069905469973783e-05 trainer/global_step‚9699 % _timestamp‚1.7884983102502487e+09Th:͵'Ç & lr-AdamW/pg1‚9.069905469973783e-05 & lr-AdamW/pg2‚9.069905469973783e-05 trainer/global_step‚9699 % _timestamp‚1.7884983102502487e+09 _runtime‚10409.890627925 _step‚796œÞpgs4¶'/ _wandbruntime‚10409  _runtime‚10409A@Ô“ ·'ùO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170224760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚662180072564176 _runtime‚10409.891912299 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚689609337730048 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97613935 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚672127839962422 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215458077 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚689610128377856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54148198 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 215458077 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9947767398246 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 % _timestamp‚1.7884983102513866e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 trainer/global_step‚9699 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9982932977152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9982932977152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚689609337730048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170224760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151762657 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚662180055524816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚764983250649600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚754574394941440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10408643917312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54148720 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215435061 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚764983038858752 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187850752 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 187850752 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚689609337730048 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97613935 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚672127839962422 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215458077 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚689610128377856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54148198 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 215458077 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9947767398246 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 % _timestamp‚1.7884983102513866e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 trainer/global_step‚9699 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9982932977152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9982932977152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚689609337730048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944I.¹'‚(:’ †£éÔ¡º® gpu.0.gpu‚17.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.835' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.166999999999998 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aºž~º'‚(:ó †£éÔô¹ï°$ memory_percent‚89.75334167480469 network.sent‚ 2.471972e+07 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚32585.60546875( disk.nvme0c0n1.out‚247042.9169921875" disk.nvme0n1.in‚32585.60546875% proc.memory.availableMB‚ 1678.8125) disk./.usagePercent‚78.88405952293958% disk./.usageGB‚1175.4294929504395 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.46484375 disk.sda.in‚0 disk.sda.out‚ 641.046875 disk.sda3.in‚0 disk.sda3.out‚ 641.046875 network.recv‚ 3.415507e+06# disk.dm-0.out‚247042.9169921875 disk.dm-1.in‚0 disk.dm-1.out‚ 641.046875# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 cpu‚25.314242929545788 proc.cpu.threads‚17 disk.sda1.out‚0& disk.nvme0n1.out‚247042.9169921875 disk.dm-0.in‚32585.60546875“¥Å 4»'/ _wandbruntime‚10410  _runtime‚10410Ò…U '¼'ƒN UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9983937722880 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151776933 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170241632 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97623583 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚754649767224320 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚679694122695168 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215479814 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 trainer/global_step‚9699 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206711808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚689678833076224 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚662246121960576 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10409875321344 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151777470 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215456772 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385721448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9948768538332 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170241634 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚765059411893760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚672194119850288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215479814 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352oT`²$½'ˆN ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚679694122695168 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215479814 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 trainer/global_step‚9699 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206711808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚689678833076224 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚662246121960576 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10409875321344 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151777470 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215456772 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385721448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9948768538332 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170241634 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚765059411893760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚672194119850288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215479814 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9983937722880 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151776933 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_þztrain_batch_end/reserved_bytes.all.current‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170241632 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97623583 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚754649767224320 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0žΔâY4¾'/ _wandbruntime‚10410  _runtime‚10410ãH‡†\¿'Ö .loss/FRt3/detection_loss‚2.807661371571677 0loss/FRt1/assignment_loss‚3.3148876269658407 /loss/FBt1/detection_loss‚2.2175043736185347 -loss/FBt3/detection_loss‚2.34444819688797 0loss/FRt2/assignment_loss‚2.6892743222415447 2loss/SRqqt1/assignment_loss‚1.3308379194140434 2loss/SRqqt2/assignment_loss‚1.6514323794593413 1loss/SRqqt4/assignment_loss‚0.733647885379579 1loss/FBt3/assignment_loss‚0.04237653921794617 1loss/FBt4/assignment_loss‚0.04411687194218071 0loss/SRqqt1/detection_loss‚2.625159751176834 1loss/SRqqt2/detection_loss‚2.2720185683833227 1loss/SRqqt3/detection_loss‚2.0640779385964074 /loss/FBt2/detection_loss‚2.2555570764975115 /loss/FRt3/assignment_loss‚2.651048000369753 /loss/FRt1/detection_loss‚2.2200230982568527 .loss/FRt2/detection_loss‚2.665699777252983 /loss/FRt4/detection_loss‚1.3624628479592502 /loss/FBt4/detection_loss‚0.5394655845538225 trainer/global_step‚9699 % _timestamp‚1.7884983112421067e+09 _runtime‚10410.882337871 &loss/total_loss‚39.221659446852456 epoch‚8 2loss/SRqqt3/assignment_loss‚1.2883586733578107 1loss/SRqqt4/detection_loss‚0.6220458123005099 _step‚799 0loss/FRt4/assignment_loss‚1.2571170762181283 0loss/FBt1/assignment_loss‚0.0725408921134374 0loss/FBt2/assignment_loss‚0.1498968631171713“aÀ'Û _runtime‚10410.882337871 &loss/total_loss‚39.221659446852456 epoch‚8 2loss/SRqqt3/assignment_loss‚1.2883586733578107 1loss/SRqqt4/detection_loss‚0.6220458123005099 _step‚799 0loss/FRt4/assignment_loss‚1.2571170762181283 0loss/FBt1/assignment_loss‚0.0725408921134374 0loss/FBt2/assignment_loss‚0.1498968631171713 .loss/FRt3/detection_loss‚2.807661371571677 0loss/FRt1/assignment_loss‚3.3148876269658407 /loss/FBt1/detection_loss‚2.2175043736185347 -loss/FBt3/detection_loss‚2.34444819688797 0loss/FRt2/assignment_loss‚2.6892743222415447 2loss/SRqqt1/assignment_loss‚1.3308379194140434 2loss/SRqqt2/assignment_loss‚1.6514323794593413 1loss/SRqqt4/assignment_loss‚0.733647885379579 1loss/FBt3/assignment_loss‚0.04237653921794617 1loss/FBt4/assignment_loss‚0.04411687194218071 0loss/SRqqt1/detection_loss‚2.625159751176834 1loss/SRqqt2/detection_loss‚2.2720185683833227 1loss/SRqqt3/detection_loss‚2.0640779385964074 /loss/FBt2/detection_loss‚2.2555570764975115 /loss/FRt3/assignment_loss‚2.651048000369753 /loss/FRt1/detection_loss‚2.2200230982568527 .loss/FRt2/detection_loss‚2.665699777252983 /loss/FRt4/detection_loss‚1.3624628479592502 /loss/FBt4/detection_loss‚0.5394655845538225 trainer/global_step‚9699 % _timestamp‚1.7884983112421067e+09Ÿë1q’Á'‚(:‡ •£éÔ„Å® gpu.0.gpu‚39.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚118.354' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉïrˆÂ'‚(:ý •£éÔ×ÃÙ± disk.dm-0.in‚32585.60546875 disk.sda3.out‚ 641.37890625) proc.memory.availableMB‚ 1677.78515625# proc.memory.rssMB‚ 2949.65234375 network.sent‚ 2.4817553e+07) disk.nvme0c0n1.out‚247467.69677734375 disk.sda3.in‚0 disk.dm-1.out‚ 641.37890625 disk.drbd1260.in‚0 disk.drbd1260.out‚1.46875 disk.sda.in‚0 disk.sda.out‚ 641.37890625" disk.nvme0n1.in‚32585.60546875$ disk.dm-0.out‚247467.69677734375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚89.75961208343506* proc.memory.percent‚18.003249168395996 cpu‚25.31393272801751 network.recv‚ 3.422168e+06) disk./.usagePercent‚78.88405952293958$ disk.nvme0c0n1.in‚32585.60546875 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1175.4294929504395' disk.nvme0n1.out‚247467.69677734375­’Ã'‚(:‡ ¤£éÔËÓ½® gpu.0.gpu‚21.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.433' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ííÐÄ'‚(:ô ¤£éÔÃ韲$ disk.dm-0.out‚247852.44677734375 disk.dm-1.out‚ 641.890625 disk.sda.in‚0) disk.nvme0c0n1.out‚247852.44677734375 disk.dm-0.in‚32585.60546875$ memory_percent‚89.76306915283203& proc.memory.availableMB‚ 1677.21875# proc.memory.rssMB‚ 2949.65234375 network.recv‚ 3.428027e+06 disk.dm-1.in‚0 disk.drbd1260.out‚1.46875 disk.sda.out‚ 641.890625* proc.memory.percent‚18.003249168395996 cpu‚25.31367258212301 proc.cpu.threads‚17 network.sent‚ 2.4915721e+07) disk./.usagePercent‚78.88563217722242 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚32585.60546875 disk.sda3.in‚0 disk.sda3.out‚ 641.890625 disk.drbd1260.in‚0% disk./.usageGB‚1175.4529266357422 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚32585.60546875' disk.nvme0n1.out‚247852.44677734375œ7Ô’Å'‚(:‡ ³£éÔú亭 gpu.0.gpu‚27.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=iŽ€Æ'‚(:õ ³£éÔ“Úå¯" disk.nvme0c0n1.in‚ 32711.640625 disk.nvme0n1.in‚ 32711.640625* proc.memory.percent‚18.003249168395996 network.sent‚ 2.4920727e+07% disk./.usageGB‚1175.4529266357422 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚93.17305088043213) proc.memory.availableMB‚ 1118.52734375 cpu‚25.313380589698628) disk./.usagePercent‚78.88563217722242 disk.dm-0.in‚ 32711.640625 disk.sda3.in‚0 disk.sda3.out‚ 642.234375 disk.dm-1.out‚ 642.234375 disk.sda.in‚0' disk.nvme0n1.out‚248421.40869140625 proc.cpu.threads‚17 network.recv‚ 3.431513e+06$ disk.dm-0.out‚248421.40869140625 disk.drbd1260.out‚ 1.47265625 disk.sda2.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚248421.40869140625# proc.memory.rssMB‚ 2949.65234375 disk.sda.out‚ 642.234375 disk.dm-1.in‚0 disk.drbd1260.in‚0)…kæ4Ç'/ _wandbruntime‚10459  _runtime‚10459v­ÈÈ' & lr-AdamW/pg2‚8.708951181757879e-05 trainer/global_step‚9749 % _timestamp‚1.7884983599349961e+09 _runtime‚10459.575380149 _step‚800 & lr-AdamW/pg1‚8.708951181757879e-05u‹xÍÉ'Ç & lr-AdamW/pg2‚8.708951181757879e-05 trainer/global_step‚9749 % _timestamp‚1.7884983599349961e+09 _runtime‚10459.575380149 _step‚800 & lr-AdamW/pg1‚8.708951181757879e-05 ©7U4Ê'/ _wandbruntime‚10459  _runtime‚10459ì(Ë'ýO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10033184677376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54412519 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚758341836020736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚675481232335530 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚693045377044480 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚683011401719296 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152508334 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚675480444647758 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216523963 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216523963 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54411999 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10460915206656 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚693045377044480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10033184677376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚768802751227392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 171068458 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 216546979 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10033957322240 _runtime‚10459.576759499 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10460716870144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10033957322240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171068458 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚9749 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚758341812080640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚693044586396672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚683011419722240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387615439 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 216546979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚768802528950784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9997837983354 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152508856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚693044586396672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884983599362078e+09 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98096335 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚665483377312816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98096337 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171068460 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _step‚801 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387592421 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9997067334942 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664`’ót(Ì'‚P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216523963 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216523963 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54411999 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10460915206656 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚693045377044480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10033184677376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚768802751227392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 171068458 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 216546979 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10033957322240 _runtime‚10459.576759499 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10460716870144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10033957322240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171068458 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚9749 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚758341812080640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚693044586396672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚683011419722240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387615439 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 216546979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚768802528950784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9997837983354 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152508856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚693044586396672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884983599362078e+09 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98096335 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚665483377312816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98096337 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171068460 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _step‚801 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387592421 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9997067334942 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10033184677376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54412519 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚758341836020736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚675481232335530 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚693045377044480 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚683011401719296 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152508334 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚675480444647758 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412¡\+„Ÿ4Í'/ _wandbruntime‚10460  _runtime‚10460-ÉÎ'‡N TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171085332 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10034962081280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216545674 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54417158 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚768878921286144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665549443748576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚802 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683079101658624 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152523141 trainer/global_step‚9749 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387631006 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10034189423104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_bÚºÐ1üatch_end/segment.all.allocated‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387654050 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 % _timestamp‚1.7884983607890902e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152523668 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171085334 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10461760862720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9998068474820 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚675547512223396 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚693114081742848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚665549460787936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9998839123440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚768879141452288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54417683 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387654050 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚758417184363520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚675548299911376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98105983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚693114081742848 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚683079119661568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683079119661568 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387631006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171085332¬‹'Ï'ŒN N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98105983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚693114081742848 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚683079119661568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683079119661568 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387631006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171085332 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171085332 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10034962081280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216545674 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54417158 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚768878921286144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665549443748576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚802 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683079101658624 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152523141 trainer/global_step‚9749 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387631006 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10034189423104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387654050 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 % _timestamp‚1.7884983607890902e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152523668 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171085334 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10461760862720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9998068474820 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚675547512223396 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚693114081742848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚665549460787936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9998839123440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚768879141452288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54417683 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387654050 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚758417184363520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚675548299911376¢'˜94Ð'/ _wandbruntime‚10460  _runtime‚10460Þ#Û(\Ñ'Ö 1loss/FBt1/assignment_loss‚0.10974722898225764 1loss/FBt2/assignment_loss‚0.13159241206117908 1loss/FBt4/assignment_loss‚0.11843315520058657 .loss/FRt4/detection_loss‚1.385984868703174 $ _timestamp‚1.788498360791562e+09 _runtime‚10460.431722582 0loss/FRt3/assignment_loss‚2.1601357575919895 .loss/FRt3/detection_loss‚2.512523240513272 /loss/FBt4/detection_loss‚0.7805412235564643 0loss/SRqqt2/detection_loss‚2.560972110146568 trainer/global_step‚9749 0loss/FRt1/assignment_loss‚2.8007308619362967 .loss/FRt1/detection_loss‚2.608723089577896 0loss/SRqqt1/detection_loss‚2.649309612553695 1loss/SRqqt3/detection_loss‚1.8690932194391887 &loss/total_loss‚38.997084430968535 _step‚803 /loss/FRt2/assignment_loss‚3.054313310805489 /loss/FBt3/detection_loss‚2.2296460308134556 epoch‚8 2loss/SRqqt3/assignment_loss‚1.2541505975774878 2loss/SRqqt4/assignment_loss‚0.8675809241093622 1loss/FBt3/assignment_loss‚0.12002067361787905 /loss/FRt2/detection_loss‚2.6198974525116263 /loss/FRt4/assignment_loss‚1.268796253372246 /loss/FBt1/detection_loss‚2.6467029651006064 1loss/SRqqt1/assignment_loss‚1.379288272097193 2loss/SRqqt2/assignment_loss‚1.2232639406408583 1loss/SRqqt4/detection_loss‚0.7586816665485944 /loss/FBt2/detection_loss‚1.8869555635111672j…çDaÒ'Û /loss/FRt2/detection_loss‚2.6198974525116263 /loss/FRt4/assignment_loss‚1.268796253372246 /loss/FBt1/detection_loss‚2.6467029651006064 1loss/SRqqt1/assignment_loss‚1.379288272097193 2loss/SRqqt2/assignment_loss‚1.2232639406408583 1loss/SRqqt4/detection_loss‚0.7586816665485944 /loss/FBt2/detection_loss‚1.8869555635111672 1loss/FBt1/assignment_loss‚0.10974722898225764 1loss/FBt2/assignment_loss‚0.13159241206117908 1loss/FBt4/assignment_loss‚0.11843315520058657 .loss/FRt4/detection_loss‚1.385984868703174 $ _timestamp‚1.788498360791562e+09 _runtime‚10460.431722582 0loss/FRt3/assignment_loss‚2.1601357575919895 .loss/FRt3/detection_loss‚2.512523240513272 /loss/FBt4/detection_loss‚0.7805412235564643 0loss/SRqqt2/detection_loss‚2.560972110146568 trainer/global_step‚9749 0loss/FRt1/assignment_loss‚2.8007308619362967 .loss/FRt1/detection_loss‚2.608723089577896 0loss/SRqqt1/detection_loss‚2.649309612553695 1loss/SRqqt3/detection_loss‚1.8690932194391887 &loss/total_loss‚38.997084430968535 _step‚803 /loss/FRt2/assignment_loss‚3.054313310805489 /loss/FBt3/detection_loss‚2.2296460308134556 epoch‚8 2loss/SRqqt3/assignment_loss‚1.2541505975774878 2loss/SRqqt4/assignment_loss‚0.8675809241093622 1loss/FBt3/assignment_loss‚0.12002067361787905£âèΈœÓ'‚(:‘ £éÔÌð‘· gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚130.15' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.030000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.ÀÌ …Ô'‚(:ú £éÔ󮹿" disk.nvme0c0n1.in‚ 32711.640625) disk.nvme0c0n1.out‚248677.64306640625 disk.drbd1260.out‚ 1.47265625 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1118.08984375 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 642.87109375# proc.memory.rssMB‚ 2949.65234375 network.recv‚ 3.435542e+06) disk./.usagePercent‚78.88564856170461 disk.sda2.in‚0 disk.nvme0n1.in‚ 32711.640625 disk.dm-0.in‚ 32711.640625 disk.dm-1.out‚ 642.87109375 cpu‚25.31307050515033 network.sent‚ 2.4925908e+07' disk.nvme0n1.out‚248677.64306640625$ disk.dm-0.out‚248677.64306640625 disk.sda3.out‚ 642.87109375 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚93.17572116851807* proc.memory.percent‚18.003249168395996% disk./.usageGB‚1175.4531707763672Àã\=Õ'‚(:’ Ñ£éÔ÷ÌЭ gpu.0.gpu‚33.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.562599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=™`|Ö'‚(:ñ Ñ£éÔõ·–±& disk.nvme0n1.out‚248838.1728515625# disk.dm-0.out‚248838.1728515625$ memory_percent‚93.17636489868164# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 643.3828125 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 32711.640625 cpu‚25.31276784825897 network.recv‚ 3.442211e+06) disk./.usagePercent‚78.88569797115872 disk.sda3.out‚ 643.3828125 disk.sda.out‚ 643.3828125" disk.nvme0c0n1.in‚ 32711.640625' proc.memory.availableMB‚ 1117.984375 disk.sda1.out‚0 network.sent‚ 2.5036766e+07 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.4765625 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚248838.1728515625% disk./.usageGB‚1175.4539070129395 disk.nvme0n1.in‚ 32711.640625†1×'‚(:’ à£éÔÜ gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.149' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.629799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òáfrØ'‚(:ç à£éÔòá׳& disk.nvme0c0n1.out‚249286.94140625 disk.dm-1.out‚ 643.765625 disk.sda2.in‚0$ memory_percent‚93.18075180053711$ disk./.usageGB‚1175.430419921875$ disk.nvme0n1.out‚249286.94140625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.4765625 disk.sda.out‚ 643.765625 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 3.446675e+06! disk.dm-0.out‚249286.94140625 disk.sda3.out‚ 643.765625 disk.sda.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 1117.265625# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 network.sent‚ 2.5042635e+07) disk./.usagePercent‚78.88412173277041" disk.nvme0c0n1.in‚ 32711.640625 disk.nvme0n1.in‚ 32711.640625 disk.dm-0.in‚ 32711.640625 disk.drbd1260.in‚0 cpu‚25.31247741737396 proc.cpu.threads‚17À§‡i4Ù'/ _wandbruntime‚10509  _runtime‚10509»K¼ªÆÚ'À % _timestamp‚1.7884984103418996e+09 _runtime‚10509.982257734 _step‚804 % lr-AdamW/pg1‚8.35443706456492e-05 % lr-AdamW/pg2‚8.35443706456492e-05 trainer/global_step‚9799‚‡ ¹ËÛ'Å % lr-AdamW/pg1‚8.35443706456492e-05 % lr-AdamW/pg2‚8.35443706456492e-05 trainer/global_step‚9799 % _timestamp‚1.7884984103418996e+09 _runtime‚10509.982257734 _step‚804¤– =4Ü'/ _wandbruntime‚10509  _runtime‚10509ìÜ­O%Ý'ÿO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚668786716140176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚678833837020866 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54674540 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚696480625711104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚686396416688640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚686396398685696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10513149854208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚696479835063296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153253275 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚696479835063296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98578735 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 _runtime‚10509.983404947 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10084209022464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚686396416688640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389548041 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389525023 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10084209022464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98578737 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10083436377600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚762109253159936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 trainer/global_step‚9799 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217612865 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232762368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10512941031936 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171912160 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171912158 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _step‚805 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217635881 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208822272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 217635881 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚668786699100816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚762109229219840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚772622170251776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 % _timestamp‚1.7884984103429463e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚686396398685696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 171912158 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153253819 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10047908568462 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚696480625711104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217612865 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54675082 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚678834624708638 [HDeviceStatsMonitor.on_train_batch_start/reÆø\»¶quested_bytes.small_pool.freed‚10047137920050 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10083436377600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚696480625711104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚686396416688640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚686396398685696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10513149854208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚696479835063296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153253275 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚696479835063296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98578735 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 _runtime‚10509.983404947 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10084209022464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚686396416688640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389548041 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389525023 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10084209022464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98578737 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10083436377600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚762109253159936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 trainer/global_step‚9799 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217612865 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232762368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10512941031936 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171912160 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171912158 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _step‚805 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217635881 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208822272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 217635881 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚668786699100816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚762109229219840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚772622170251776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 % _timestamp‚1.7884984103429463e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚686396398685696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 171912158 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153253819 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10047908568462 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489¥¤ãUu4ß'/ _wandbruntime‚10510  _runtime‚10510Ý©Þ'à'‰N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389586652 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚686464116627968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217634576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54680396 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚762184577562624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10048909708548 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171929032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171929032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10514186343936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚686464116627968 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153268781 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚696548539748352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚686464098625024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚772698567680512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10084441123328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚772698787846656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54679863 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚762184601502720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10513990117888 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚678901692284484 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10085213781504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389586652 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚10510.877466715 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217657618 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚678900904596504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚696549330409472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚668852765536576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171929034 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9799 _step‚806 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚696549330409472 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153268246 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489{óÕ'á'ŽN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171929034 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9799 _step‚806 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚696549330409472 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153268246 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389586652 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚686464116627968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217634576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54680396 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚762184577562624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10048909708548 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171929032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171929032 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10514186343936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚686464116627968 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153268781 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚696548539748352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚686464098625024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚772698567680512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10084441123328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚772698787846656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54679863 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚762184601502720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10513990117888 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚678901692284484 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10085213781504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389586652 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚10510.877466715 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217657618 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚678900904596504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚696549330409472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚668852765536576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664¦)S,ø4â'/ _wandbruntime‚10510  _runtime‚10510¦´ÔHTã'Î 0loss/FRt4/assignment_loss‚1.0499597330888113 -loss/FBt2/assignment_loss‚0.2794122343862 -loss/FRt2/detection_loss‚2.73184406766948 1loss/SRqqt3/detection_loss‚2.1695901471618058 epoch‚8 -loss/FRt3/detection_loss‚3.22714624715888 0loss/SRqqt1/detection_loss‚2.223085037432611 1loss/SRqqt4/detection_loss‚0.7581741340162064 _runtime‚10510.879249916 _step‚807 0loss/SRqqt2/detection_loss‚2.299977988998095 /loss/FBt4/detection_loss‚0.5392560669154461 2loss/SRqqt3/assignment_loss‚1.5147028413374681 2loss/SRqqt4/assignment_loss‚0.8564097180602733 % _timestamp‚1.7884984112391057e+09 2loss/SRqqt1/assignment_loss‚1.3860526296775788 1loss/FBt3/assignment_loss‚0.18655436033440628 /loss/FRt1/detection_loss‚2.2351221699267625 /loss/FRt4/detection_loss‚1.4388096720601122 .loss/FBt2/detection_loss‚2.426240700483322 &loss/total_loss‚39.495895877440375 0loss/FBt1/assignment_loss‚0.2715203338514336 trainer/global_step‚9799 .loss/FRt3/assignment_loss‚2.57289893730827 1loss/SRqqt2/assignment_loss‚1.498198447500666 .loss/FBt1/detection_loss‚2.293827094814994 .loss/FBt3/detection_loss‚2.475280980269114 0loss/FRt1/assignment_loss‚2.4027286604046822 0loss/FRt2/assignment_loss‚2.5907838304837543 1loss/FBt4/assignment_loss‚0.06831984410000488ot)ä'Ó .loss/FRt3/assignment_loss‚2.57289893730827 1loss/SRqqt2/assignment_loss‚1.498198447500666 .loss/FBt1/detection_loss‚2.293827094814994 .loss/FBt3/detection_loss‚2.475280980269114 0loss/FRt1/assignment_loss‚2.4027286604046822 0loss/FRt2/assignment_loss‚2.590783830483”0 ‰07543 1loss/FBt4/assignment_loss‚0.06831984410000488 0loss/FRt4/assignment_loss‚1.0499597330888113 -loss/FBt2/assignment_loss‚0.2794122343862 -loss/FRt2/detection_loss‚2.73184406766948 1loss/SRqqt3/detection_loss‚2.1695901471618058 epoch‚8 -loss/FRt3/detection_loss‚3.22714624715888 0loss/SRqqt1/detection_loss‚2.223085037432611 1loss/SRqqt4/detection_loss‚0.7581741340162064 _runtime‚10510.879249916 _step‚807 0loss/SRqqt2/detection_loss‚2.299977988998095 /loss/FBt4/detection_loss‚0.5392560669154461 2loss/SRqqt3/assignment_loss‚1.5147028413374681 2loss/SRqqt4/assignment_loss‚0.8564097180602733 % _timestamp‚1.7884984112391057e+09 2loss/SRqqt1/assignment_loss‚1.3860526296775788 1loss/FBt3/assignment_loss‚0.18655436033440628 /loss/FRt1/detection_loss‚2.2351221699267625 /loss/FRt4/detection_loss‚1.4388096720601122 .loss/FBt2/detection_loss‚2.426240700483322 &loss/total_loss‚39.495895877440375 0loss/FBt1/assignment_loss‚0.2715203338514336 trainer/global_step‚9799§G,>Q’å'‚(:‡ ï£éÔÅòÝ® gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.436' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®N5¢‚æ'‚(:÷ ï£éÔºä°* proc.memory.percent‚18.003249168395996 cpu‚25.312190881939785 network.sent‚ 2.5058863e+07) disk./.usagePercent‚78.88412173277041 disk.sda.out‚ 644.1015625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2949.65234375 proc.cpu.threads‚17 network.recv‚ 3.450049e+06$ disk./.usageGB‚1175.430419921875) disk.nvme0c0n1.out‚249904.21337890625! disk.nvme0n1.in‚ 32871.7265625 disk.dm-0.in‚ 32871.7265625 disk.sda3.out‚ 644.1015625 disk.dm-1.in‚0& proc.memory.availableMB‚ 1118.09375# disk.nvme0c0n1.in‚ 32871.7265625' disk.nvme0n1.out‚249904.21337890625$ disk.dm-0.out‚249904.21337890625 disk.dm-1.out‚ 644.1015625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.48046875 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0$ memory_percent‚93.17569732666016sF…’ç'‚(:‡ þ£éԘ܅® gpu.0.gpu‚45.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.037' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8074 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íÎ\C{è'‚(:ð þ£éԥ㊲ network.sent‚ 2.5155598e+07$ disk./.usageGB‚1175.453857421875 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 32871.7265625# disk.nvme0n1.out‚250160.6015625 network.recv‚ 3.455554e+06 disk.sda3.in‚0 disk.sda3.out‚ 644.62109375 disk.drbd1260.in‚0$ memory_percent‚93.17879676818848( proc.memory.availableMB‚ 1117.5859375# proc.memory.rssMB‚ 2949.65234375 proc.cpu.threads‚17 disk.sda1.out‚0! disk.nvme0n1.in‚ 32871.7265625 disk.dm-1.in‚0 disk.dm-1.out‚ 644.62109375 disk.drbd1260.out‚ 1.48046875 disk.sda.in‚0) disk./.usagePercent‚78.88569464306077 disk.sda2.out‚0% disk.nvme0c0n1.out‚250160.6015625 disk.sda.out‚ 644.62109375* proc.memory.percent‚18.003249168395996 cpu‚25.311908106564506 disk.dm-0.in‚ 32871.7265625 disk.dm-0.out‚250160.6015625R¨é$’é'‚(:‡ ¤éÔÿ½Ö¯ gpu.0.gpu‚36.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.041' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 "þé€ê'‚(:õ ¤éÔ÷ÉȲ$ memory_percent‚93.17727088928223 network.recv‚ 3.459465e+06) disk./.usagePercent‚78.88569669112105 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 32871.7265625' disk.nvme0n1.out‚250585.35888671875# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17 disk.sda3.out‚ 645.015625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 32871.7265625 cpu‚25.311630117170917 network.sent‚ 2.5159707e+07 disk.sda3.in‚0) disk.nvme0c0n1.out‚250585.35888671875$ disk.dm-0.out‚250585.35888671875( proc.memory.availableMB‚ 1117.8359375% disk./.usageGB‚1175.4538879394531 disk.dm-1.in‚0 disk.dm-1.out‚ 645.015625 disk.drbd1260.out‚1.484375 disk.sda.out‚ 645.015625 disk.sda1.out‚0! disk.nvme0n1.in‚ 32871.7265625—üß2’ë'‚(:‡ œ¤éÔùÖÊ­ gpu.0.gpu‚49.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:¼V…ì'‚(:ú œ¤éÔ³¿‡² disk.sda1.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 32871.7265625! disk.nvme0n1.in‚ 32871.7265625$ disk.dm-0.out‚250869.55419921875) proc.memory.availableMB‚ 1117.83203125 disk.dm-1.in‚0 disk.sda.out‚ 645.48046875' disk.nvme0n1.out‚250920.80419921875 disk.dm-0.in‚ 32871.7265625# proc.memory.rssMB‚ 2949.65234375) disk./.usagePercent‚78.88569669112105 disk.sda3.in‚0 disk.drbd1260.out‚1.484375$ memory_percent‚93.17729473114014* proc.memory.percent‚18.003249168395996 cpu‚25.31133427833546 proc.cpu.threads‚17 network.recv‚ 3.463778e+06% disk./.usageGB‚1175.4538879394531 disk.sda3.out‚ 645.48046875 disk.dm-1.out‚ 645.48046875 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚250920.80419921875 network.sent‚ 2.516674e+07 disk.sda1.in‚0AÆ4í'/ _wandbruntime‚10561  _runtime‚10561Ï^ÜWÇî'Á $ _timestamp‚1.788498461464573e+09 _runtime‚10561.104997641 _step‚808 & lr-AdamW/pg1‚8.006436998605594e-05 & lr-AdamW/pg2‚8.006436998605594e-05 trainer/global_step‚9849”îgÞÌï'Æ & lr-AdamW/pg1‚8.006436998605594e-05 & lr-AdamW/pg2‚8.006436998605594e-05 trainer/global_step‚9849 $ _timestamp‚1.788498461464573e+09 _runtime‚10561.104997641 _step‚808¨:¨‘`4ð'/ _wandbruntime‚10561  _runtime‚10561¶×T(ñ'üO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10097208505158 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54939438 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚699915874377728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚546 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 " _timestamp‚1.7884984614657e+09 _step‚809 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚776441704041472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 trainer/global_step‚9849 _runtime‚10561.106170622 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚682187229393974 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚672090037928176 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172755860 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚689781395652096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172755858 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚699915083729920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10565262310400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10097979153570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10134460722688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚699915874377728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚689781395652096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99061137 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 391480643 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172755858 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154000575 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚765876646359040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10565057682432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚689781413655040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚672090020888816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10133688077824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚689781413655040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154000027 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚699915083729920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10133688077824 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218701767 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 J(ò'P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218701767 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 " _timestamp‚1.7884984614657e+09 _step‚809 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚776441704041472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 trainer/global_step‚9849 _runtime‚10561.106170622 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚682187229393974 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚672090037928176 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172755860 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚689781395652096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172755858 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚699915083729920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10565262310400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10097979153570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10134460722688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚699915874377728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚689781395652096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99061137 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 391480643 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172755858 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154000575 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚765876646359040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10565057682432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚689781413655040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚672090020888816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10133688077824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚689781413655040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154000027 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚699915083729920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10133688077824©κ£å4ó'/ _wandbruntime‚10561  _runtime‚10561áDç^×ô'ˆN T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚699984579076096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚538 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚689849113594368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 391496210 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218746520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10566114339840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10098209645036 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99070785 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚689849113594368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚672156104363936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54944756 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚682254296969612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10134692823552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154015541 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚682255084657592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚810 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218746520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚765952018641920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚10561.960914436 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172772734 L>DeviceSt(â8«7atsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391519254 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218723478 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10098980293656 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 391496210 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10135465481728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99070783 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚776518109041664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172772734 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚699983788414976 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154015001 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚672156087324576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚699984579076096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10134692823552 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 $ _timestamp‚1.788498462319599e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚765951994701824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664&nmÕ'õ'N $ _timestamp‚1.788498462319599e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚765951994701824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚699984579076096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚538 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚689849113594368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 391496210 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218746520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10566114339840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10098209645036 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99070785 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚689849113594368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚672156104363936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54944756 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚682254296969612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10134692823552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154015541 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚682255084657592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚810 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218746520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚765952018641920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 _runtime‚10561.960914436 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172772734 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391519254 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218723478 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10098980293656 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 391496210 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10135465481728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99070783 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚776518109041664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172772734 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚699983788414976 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154015001 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚672156087324576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚699984579076096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10134692823552 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005ª“‹„±4ö'/ _wandbruntime‚10561  _runtime‚10561”b.X^÷'Ø /loss/FRt4/assignment_loss‚0.798003011766602 1loss/FBt3/assignment_loss‚0.15328222114474796 1loss/FBt4/assignment_loss‚0.19166564884419351 .loss/FRt2/detection_loss‚2.572092511105666 _step‚811 2loss/SRqqt2/assignment_loss‚1.4204331995630315 .loss/FRt1/detection_loss‚2.310061517260259 1loss/SRqqt2/detection_loss‚2.3657242357730865 _runtime‚10561.962478143 0loss/FRt1/assignment_loss‚2.9527523545991805 2loss/SRqqt3/assignment_loss‚1.3144800754492743 2loss/SRqqt4/assignment_loss‚0.8916090641709927 0loss/FBt2/assignment_loss‚0.2650306112142668 /loss/FRt3/detection_loss‚2.9393721305090805 /loss/FBt2/detection_loss‚2.5891434152921042 epoch‚8 % _timestamp‚1.7884984623222797e+09 0loss/FRt2/assignment_loss‚2.8019572611038503 0loss/SRqqt1/detection_loss‚2.177022497500143 0loss/FBt4/detection_loss‚0.41979973659390346 trainer/global_step‚9849 /loss/FRt4/detection_loss‚1.5319799209999687 1loss/SRqqt4/detection_loss‚0.7786668428268871 /loss/FBt3/detection_loss‚2.2805507050620184 0loss/FRt3/assignment_loss‚2.4298311369172456 1loss/SRqqt1/assignment_loss‚1.257893416309549 0loss/FBt1/assignment_loss‚0.4781902480042635 &loss/total_loss‚39.046362647613854 0loss/SRqqt3/detection_loss‚1.577877293161133 /loss/FBt1/detection_loss‚2.5489435924424066»#ͧcø'Ý % _timestamp‚1.7884984623222797e+09 0loss/FRt2/assignment_loss‚2.8019572611038503 0loss/SRqqt1/detection_loss‚2.177022497500143 0loss/FBt4/detection_loss‚0.41979973659390346 trainer/global_step‚9849 /loss/FRt4/detection_loss‚1.5319799209999687 1loss/SRqqt4/detection_loss‚0.7786668428268871 /loss/FBt3/detection_loss‚2.2805507050620184 0loss/FRt3/assignment_loss‚2.4298311369172456 1loss/SRqqt1/assignment_loss‚1.257893416309549 0loss/FBt1/assignment_loss‚0.4781902480042635 &loss/total_loss‚39.046362647613854 0loss/SRqqt3/detection_loss‚1.577877293161133 /loss/FBt1/detection_loss‚2.5489435924424066 /loss/FRt4/assignment_loss‚0.798003011766602 1loss/FBt3/assignment_loss‚0.15328222114474796 1loss/FBt4/assignment_loss‚0.19166564884419351 .loss/FRt2/detection_loss‚2.572092511105666 _step‚811 2loss/SRqqt2/assignment_loss‚1.4204331995630315 .loss/FRt1/detection_loss‚2.310061517260259 1loss/SRqqt2/detection_loss‚2.3657242357730865 _runtime‚10561.962478143 0loss/FRt1/assignment_loss‚2.9527523545991805 2loss/SRqqt3/assignment_loss‚1.3144800754492743 2loss/SRqqt4/assignment_loss‚0.8916090641709927 0loss/FBt2/assignment_loss‚0.2650306112142668 /loss/FRt3/detection_loss‚2.9393721305090805 /loss/FBt2/detection_loss‚2.5891434152921042 epoch‚8«l#>kù'‚(:… «¤éÔ¦êû± gpu.0.gpu‚35.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.27' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J©¡ …ú'‚(:ú «¤éÔÜ郴 disk.drbd1260.in‚0 disk.sda2.out‚0# memory_percent‚93.1694746017456) proc.memory.availableMB‚ 1119.11328125* proc.memory.percent‚18.003249168395996 cpu‚25.311050658458836 proc.cpu.threads‚17 network.sent‚ 2.5170069e+07% disk./.usageGB‚1175.4304084777832# disk.nvme0c0n1.in‚ 32979.9140625! disk.nvme0n1.in‚ 32980.0390625 disk.dm-1.in‚0 disk.sda1.out‚0 network.recv‚ 3.467287e+06) disk./.usagePercent‚78.88412096474781 disk.sda3.in‚0 disk.sda3.out‚ 645.76171875 disk.sda.out‚ 645.76171875# proc.memory.rssMB‚ 2949.65234375 disk.dm-1.out‚ 645.76171875( disk.nvme0c0n1.out‚251057.6650390625& disk.nvme0n1.out‚251057.6650390625 disk.dm-0.in‚ 32980.0390625 disk.drbd1260.out‚ 1.48828125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0# disk.dm-0.out‚251057.6650390625—+s’û'‚(:‡ º¤éÔ£Éê® gpu.0.gpu‚46.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.281' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0562 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02<—=ˆü'‚(:ý º¤éÔŠ£Ú° disk.sda1.in‚0# proc.memory.rssMB‚ 2949.65234375 network.sent‚ 2.5277845e+07% disk./.usageGB‚1175.4305000305176( disk.nvme0c0n1.out‚251530.3017578125# disk.dm-0.out‚251530.3017578125 disk.dm-1.out‚ 646.16796875 network.recv‚ 3.472416e+06 disk.sda2.in‚0" disk.nvme0n1.in‚32987.50390625& disk.nvme0n1.out‚251530.3017578125 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.48828125 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 646.16796875 disk.sda1.out‚0$ memory_percent‚93.17095279693604) proc.memory.availableMB‚ 1118.87109375* proc.memory.percent‚18.003249168395996 cpu‚25.31077003482363 proc.cpu.threads‚17) disk./.usagePercent‚78.88412710892862 disk.sda2.out‚0$ disk.nvme0c0n1.in‚32987.50390625 disk.dm-0.in‚32987.50390625 disk.sda3.out‚ 646.16796875 }f–’ý'‚(:‡ ɤéÔŽÁÿ­ gpu.0.gpu‚45.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.328' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2656 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úš0‡þ'‚(:ü ɤéÔæ®’° network.recv‚ 3.477964e+06) disk./.usagePercent‚78.81226733011259 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.sda.in‚0$ memory_percent‚93.16942691802979 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚32987.50390625 disk.sda3.in‚0 disk.sda3.out‚ 646.9453125 disk.dm-1.out‚ 646.9453125# proc.memory.rssMB‚ 2949.65234375 network.sent‚ 2.5284556e+07) disk.nvme0c0n1.out‚252062.28076171875" disk.nvme0n1.in‚32987.50390625 disk.dm-0.in‚32987.50390625 disk.drbd1260.out‚ 1.4921875 disk.sda.out‚ 646.9453125) proc.memory.availableMB‚ 1119.12109375 cpu‚25.31045357503666% disk./.usageGB‚1174.3597373962402' disk.nvme0n1.out‚252062.28076171875$ disk.dm-0.out‚252062.28076171875 disk.dm-1.in‚0* proc.memory.percent‚18.0032491683959966g*4ÿ'/ _wandbruntime‚10610  _runtime‚10610ÈúýÈ€( % _timestamp‚1.7884985113523746e+09 _runtime‚10610.992760377 _step‚812 & lr-AdamW/pg1‚7.665023506572054e-05 & lr-AdamW/pg2‚7.665023506572054e-05 trainer/global_step‚9899T'Í(Ç _runtime‚10610.992760377 _step‚812 & lr-AdamW/pg1‚7.665023506572054e-05 & lr-AdamW/pg2‚7.665023506572054e-05 trainer/global_step‚9899 % _timestamp‚1.7884985113523746e+09¬ Kl 4‚(/ _wandbruntime‚10610  _runtime‚10610Á|x(ƒ(ÿO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703350332396544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚703350332396544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚693166392618496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99543535 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚769644063498240 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10148049738678 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 393413245 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173599560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206725120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10184712422912 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154748078 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚685540621767082 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚813 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚685541409454854 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219813685 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10147279090266 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55204541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55203982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173599560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10184712422912 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚675393359716176 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚780261359791104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚693166392618496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230665216 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10617296292864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393390227 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219813685 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚703351123044352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚675393342676816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10183939778048 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚693166392618496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230665216 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10617296292864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393390227 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219813685 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚703351123044352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚675393342676816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10183939778048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703350332396544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚703350332396544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚693166392618496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99543535 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚769644063498240 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10148049738678 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 393413245 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173599560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206725120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10184712422912 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154748078 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚685540621767082 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚813 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚685541409454854 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219813685 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10147279090266 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55204541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55203982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173599560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10184712422912 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚675393359716176 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚780261359791104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 B(šÓ4…(/ _wandbruntime‚10611  _runtime‚10611;aŒN'†(‰N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚554 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚780337976818688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚675459426151936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99553185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚693234110560768 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚693234092557824 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154762985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚769719411841024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393451856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219835422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99553183 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 173616432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10148280230144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 % _timestamp‚1.7884985122287493e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10618342714368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚675459409112576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 393428812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154762431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55209248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚693234110560768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219812380 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚685607689342720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10149050878764 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ízÌA'‡(ŽN ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚675459426151936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99553185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚693234110560768 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚693234092557824 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154762985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚769719411841024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393451856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219835422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99553183 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 173616432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10148280230144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 % _timestamp‚1.7884985122287493e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10618342714368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚675459409112576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 393428812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154762431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55209248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚693234110560768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219812380 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚685607689342720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10149050878764 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚554 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚780337976818688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39®¾šÿ`4ˆ(/ _wandbruntime‚10611  _runtime‚10611)èÎía‰(Û 2loss/SRqqt1/assignment_loss‚1.3710452561577162 0loss/SRqqt1/detection_loss‚2.438400731484095 % _timestamp‚1.7884985122310367e+09 2loss/SRqqt4/assignment_loss‚0.9049056539635943 3loss/FBt3/assignment_loss‚0.0007832882474628396 1loss/SRqqt4/detection_loss‚1.0128586293481614 epoch‚8 trainer/global_step‚9899 /loss/FRt1/assignment_loss‚2.654509211128408 0loss/FRt3/assignment_loss‚2.5279004554594717 /loss/FRt3/detection_loss‚2.9250142802634547 /loss/FRt4/detection_loss‚1.5093765749192949 /loss/FBt1/detection_loss‚2.2953795875821794 .loss/FBt2/detection_loss‚2.519002695878347 1loss/SRqqt2/detection_loss‚1.9529730159985392 1loss/SRqqt3/detection_loss‚1.7230120401116127 .loss/FBt3/detection_loss‚2.976496994495392 0loss/FBt4/detection_loss‚0.41217715728361115 _runtime‚10611.871187391 /loss/FRt2/assignment_loss‚3.216239633304732 0loss/FRt4/assignment_loss‚1.3867464617117127 2loss/SRqqt3/assignment_loss‚1.4035669356743072 $loss/total_loss‚40.3738550271021 _step‚815 2loss/FBt1/assignment_loss‚0.021427435592093907 .loss/FRt1/detection_loss‚2.628969390622594 2loss/SRqqt2/assignment_loss‚1.5856834691213935 1loss/FBt2/assignment_loss‚0.13220766873979528 1loss/FBt4/assignment_loss‚0.11606423221250237 /loss/FRt2/detection_loss‚2.6591142278016338Ú÷ffŠ(à 2loss/FBt1/assignment_loss‚0.021427435592093907 .loss/FRt1/detection_loss‚2.628969390622594 2loss/SRqqt2/assignment_loss‚1.5856834691213935 1loss/FBt2/assignment_loss‚0.13220766873979528 1loss/FBt4/assignment_loss‚0.11606423221250237 /loss/FRt2/detection_loss‚2.6591142278016338 2loss/SRqqt1/assignment_loss‚1.3710452561577162 0loss/SRqqt1/detection_loss‚2.438400731484095 % _timestamp‚1.7884985122310367e+09 2loss/SRqqt4/assignment_loss‚0.9049056539635943 3loss/FBt3/assignment_loss‚0.0007832882474628396 1loss/SRqqt4/detection_loss‚1.0128586293481614 epoch‚8 trainer/global_step‚9899 /loss/FRt1/assignment_loss‚2.654509211128408 0loss/FRt3/assignment_loss‚2.5279004554594717 /loss/FRt3/detection_loss‚2.9250142802634547 /loss/FRt4/detection_loss‚1.5093765749192949 /loss/FBt1/detection_loss‚2.2953795875821794 .loss/FBt2/detection_loss‚2.519002695878347 1loss/SRqqt2/detection_loss‚1.9529730159985392 1loss/SRqqt3/detection_loss‚1.7230120401116127 .loss/FBt3/detection_loss‚2.976496994495392 0loss/FBt4/detection_loss‚0.41217715728361115 _runtime‚10611.871187391 /loss/FRt2/assignment_loss‚3.216239633304732 0loss/FRt4/assignment_loss‚1.3867464617117127 2loss/SRqqt3/assignment_loss‚1.4035669356743072 $loss/total_loss‚40.3738550271021 _step‚815¯ˆ·x©l‹(‚(:† ؤéÔü…½¯ gpu.0.gpu‚4.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.7949981689453mF¸>%1* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.808' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸àzɇŒ(‚(:ü ؤéÔÅâË° disk.dm-0.in‚32987.50390625 disk.dm-1.out‚ 647.4296875 disk.sda.in‚0 disk.sda.out‚ 647.4296875 disk.sda1.out‚0" disk.nvme0n1.in‚32987.50390625# proc.memory.rssMB‚ 2949.65234375 network.sent‚ 2.5290426e+07 network.recv‚ 3.482396e+06 disk.sda3.out‚ 647.4296875 disk.dm-1.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚32987.50390625$ memory_percent‚93.17116737365723* proc.memory.percent‚18.003249168395996% disk./.usageGB‚1174.3836936950684$ disk.dm-0.out‚252615.36669921875 disk.sda3.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.4921875 disk.sda1.in‚0) disk.nvme0c0n1.out‚252615.36669921875' disk.nvme0n1.out‚252615.36669921875( proc.memory.availableMB‚ 1118.8359375 cpu‚25.310179627361126 proc.cpu.threads‚17) disk./.usagePercent‚78.81387505742761¬†æª’(‚(:‡ ç¤éÔµÿᬠgpu.0.gpu‚22.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚167.286' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07é:ƇŽ(‚(:ü ç¤éÔ™·°® network.recv‚ 3.487686e+06 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚32987.50390625$ memory_percent‚93.16983222961426 proc.cpu.threads‚17 disk.dm-1.out‚ 647.63671875 disk.drbd1260.out‚ 1.49609375 disk.sda.out‚ 647.63671875 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚253000.5361328125 disk.dm-0.in‚32987.50390625( proc.memory.availableMB‚ 1119.0546875 network.sent‚ 2.5398588e+07% disk./.usageGB‚1173.0421104431152 disk.sda3.in‚0# disk.dm-0.out‚253000.5361328125# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 cpu‚25.309849719118162( disk./.usagePercent‚78.7238402797257 disk.sda3.out‚ 647.63671875 disk.drbd1260.in‚0 disk.sda.in‚0( disk.nvme0c0n1.out‚253000.5361328125" disk.nvme0n1.in‚32987.50390625è¡œ(‚(:’ ö¤éÔíض° gpu.0.gpu‚41.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.118' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.023599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·>½,€(‚(:õ ö¤éÔàìܱ disk.sda.in‚0( disk.nvme0c0n1.out‚253259.3369140625 disk.dm-0.in‚33120.55078125# disk.dm-0.out‚253259.3369140625$ memory_percent‚93.16983222961426* proc.memory.percent‚18.003249168395996 network.sent‚ 2.5403671e+07 network.recv‚ 3.491188e+06 disk.drbd1260.out‚ 1.49609375 disk.sda.out‚ 648.078125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1119.0546875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0" disk.nvme0n1.in‚33120.55078125& disk.nvme0n1.out‚253259.3369140625 cpu‚25.30951668065406 proc.cpu.threads‚17( disk./.usagePercent‚78.7345334584234% disk./.usageGB‚1173.2014465332031 disk.sda3.out‚ 648.078125$ disk.nvme0c0n1.in‚33120.55078125# proc.memory.rssMB‚ 2949.65234375 disk.dm-1.out‚ 648.078125CN£4‘(/ _wandbruntime‚10658  _runtime‚10658ã*è`Ä’(¾ % _timestamp‚1.7884985587729416e+09 _runtime‚10658.413272141 _step‚816 $ lr-AdamW/pg1‚7.3302677385243e-05 $ lr-AdamW/pg2‚7.3302677385243e-05 trainer/global_step‚9949Çl žÉ“(à $ lr-AdamW/pg2‚7.3302677385243e-05 trainer/global_step‚9949 % _timestamp‚1.7884985587729416e+09 _runtime‚10658.413272141 _step‚816 $ lr-AdamW/pg1‚7.3302677385243e-05°¾i÷4”(/ _wandbruntime‚10658  _runtime‚10658Ϋ¨µ(•(P KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55469484 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚706786371710976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚696551407587840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚784081263204352 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884985587740228e+09 _runtime‚10658.414581419 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220879571 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155495982 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10669558193152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10234191478272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _step‚817 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚773411480637440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10198120323786 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚706785581063168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220879571 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚773411504577536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10669758626816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚696551389584896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688894801827962 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚696551389584896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10234964123136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚784081038830592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992G-%² (–(†P OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚773411480637440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10198120323786 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚706785581063168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220879571 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚773411504577536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10669758626816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚696551389584896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚688894801827962 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚696551389584896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10234964123136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚784081038830592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55469484 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚706786371710976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚696551407587840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚784081263204352 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 % _timestamp‚1.7884985587740228e+09 _runtime‚10658.414581419 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220879571 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155495982 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10669558193152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10234191478272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _step‚817±â†š4—(/ _wandbruntime‚10659  _runtime‚10659SA^„€˜(‹N ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10235968882176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚9949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220901282 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174460132 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10235196224000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100035583 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚696619089524224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚773486852920320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚706855076409344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174460132 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220901282 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚678762730900576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220924324 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678762747939936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚706854285748224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55475227 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183643136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10235968882176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174460134 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚784157668631552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10198350815252 % _timestamp‚1.7884985597123666e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395361414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220924324 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395384458 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155510284 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚10659.353314147 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _step‚818 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚706855076409344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 REDeviceStatsMonitor.on_train_batch_end/ina]2Ó¾‘ctive_split.small_pool.freed‚55474701 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚696619107527168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚696619107527168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155510812 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10670632068096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55475227 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 183643136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10235968882176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174460134 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚784157668631552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10198350815252 % _timestamp‚1.7884985597123666e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395361414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220924324 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395384458 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155510284 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚10659.353314147 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _step‚818 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚706855076409344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55474701 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚696619107527168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚696619107527168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155510812 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10670632068096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10235196224000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100035583 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚696619089524224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚773486852920320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚706855076409344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174460132 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220901282 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚678762730900576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220924324 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678762747939936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚706854285748224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846²b­ã)4š(/ _wandbruntime‚10659  _runtime‚10659$YE€^›(Ø 0loss/FRt1/assignment_loss‚3.2429830567403273 2loss/SRqqt4/assignment_loss‚0.9800261547681929 .loss/FRt1/detection_loss‚2.147066696716303 epoch‚8 1loss/FBt1/assignment_loss‚0.14815514294117646 /loss/FRt3/detection_loss‚2.7972187338204217 1loss/SRqqt1/detection_loss‚2.2141398921095092 /loss/FRt2/assignment_loss‚2.593577053617029 /loss/FRt4/detection_loss‚1.4227244709703055 /loss/FBt2/detection_loss‚2.3555001345547764 2loss/SRqqt3/assignment_loss‚1.2694584376045637 0loss/FBt3/assignment_loss‚0.0825515793156228 0loss/FRt4/assignment_loss‚0.8987312869592147 1loss/SRqqt2/detection_loss‚2.5405045092105865 /loss/FBt1/detection_loss‚2.6185116602314844 /loss/FBt4/detection_loss‚0.4912076957213382 trainer/global_step‚9949 _step‚819 %loss/total_loss‚39.33184356798918 1loss/SRqqt1/assignment_loss‚1.311128603198148 1loss/FBt2/assignment_loss‚0.10386164555680724 .loss/FRt2/detection_loss‚2.708705144837472 1loss/SRqqt4/detection_loss‚0.8981244837030254 .loss/FBt3/detection_loss‚2.518968391418457 0loss/FRt3/assignment_loss‚2.3212657344752343 1loss/SRqqt3/detection_loss‚1.9932851333703314 2loss/SRqqt2/assignment_loss‚1.5938924907147884 1loss/FBt4/assignment_loss‚0.08025543543406753 % _timestamp‚1.7884985597146358e+09 _runtime‚10659.354801911ØÖ±½cœ(Ý /loss/FRt2/assignment_loss‚2.593577053617029 /loss/FRt4/detection_loss‚1.4227244709703055 /loss/FBt2/detection_loss‚2.3555001345547764 2loss/SRqqt3/assignment_loss‚1.2694584376045637 0loss/FBt3/assignment_loss‚0.0825515793156228 0loss/FRt4/assignment_loss‚0.8987312869592147 1loss/SRqqt2/detection_loss‚2.5405045092105865 /loss/FBt1/detection_loss‚2.6185116602314844 /loss/FBt4/detection_loss‚0.4912076957213382 trainer/global_step‚9949 _step‚819 %loss/total_loss‚39.33184356798918 1loss/SRqqt1/assignment_loss‚1.311128603198148 1loss/FBt2/assignment_loss‚0.10386164555680724 .loss/FRt2/detection_loss‚2.708705144837472 1loss/SRqqt4/detection_loss‚0.8981244837030254 .loss/FBt3/detection_loss‚2.518968391418457 0loss/FRt3/assignment_loss‚2.3212657344752343 1loss/SRqqt3/detection_loss‚1.9932851333703314 2loss/SRqqt2/assignment_loss‚1.5938924907147884 1loss/FBt4/assignment_loss‚0.08025543543406753 % _timestamp‚1.7884985597146358e+09 _runtime‚10659.354801911 0loss/FRt1/assignment_loss‚3.2429830567403273 2loss/SRqqt4/assignment_loss‚0.9800261547681929 .loss/FRt1/detection_loss‚2.147066696716303 epoch‚8 1loss/FBt1/assignment_loss‚0.14815514294117646 /loss/FRt3/detection_loss‚2.7972187338204217 1loss/SRqqt1/detection_loss‚2.2141398921095092³NoÒ'(‚(:’ …¥éÔÇùʱ gpu.0.gpu‚48.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.149' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.629799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æT‚ž(‚(:÷ …¥éÔ´ì³ network.recv‚ 3.495839e+06) disk./.usagePercent‚78.73295850007274$ disk./.usageGB‚1173.177978515625$ disk.nvme0c0n1.in‚33120.55078125$ disk.dm-0.out‚253427.13818359375 disk.sda3.out‚ 649.00390625 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 649.00390625 disk.sda.in‚0 disk.sda.out‚ 649.00390625 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚93.17002296447754# proc.memory.rssMB‚ 2949.65234375) disk.nvme0c0n1.out‚253427.13818359375" disk.nvme0n1.in‚33120.55078125' disk.nvme0n1.out‚253427.13818359375 disk.dm-0.in‚33120.55078125 disk.sda3.in‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.sent‚ 2.5409837e+07 disk.drbd1260.out‚1.5( proc.memory.availableMB‚ 1119.0234375* proc.memory.percent‚18.003249168395996 cpu‚25.30919411766072µÌ-ñ’Ÿ(‚(:‡ ”¥éÔõŸô¬ gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.281' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0562 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßmz (‚(:ï ”¥éÔ¤ÊЮ# proc.memory.rssMB‚ 2949.65234375 cpu‚25.30885468176229 proc.cpu.threads‚17$ disk./.usageGB‚1173.091537475586 disk.dm-1.in‚0" disk.nvme0n1.in‚33120.55078125 disk.dm-0.in‚33120.55078125) proc.memory.availableMB‚ 1119.01171875* proc.memory.percent‚18.003249168395996) disk./.usagePercent‚78.72715736934687 disk.sda3.in‚0 disk.dm-1.out‚ 649.40625 disk.sda.out‚ 649.40625 disk.sda2.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚253762.93505859375$ disk.dm-0.out‚253762.93505859375 network.recv‚ 3.502344e+06 disk.sda3.out‚ 649.40625 disk.drbd1260.in‚0 disk.drbd1260.out‚1.5 disk.sda.in‚0 disk.sda1.in‚0' disk.nvme0n1.out‚253762.93505859375$ memory_percent‚93.17009449005127 network.sent‚ 2.5518657e+07 disk.sda1.out‚0$ disk.nvme0c0n1.in‚33120.55078125nï (¡(‚(:’ £¥éÔÅÕŠ­ gpu.0.gpu‚48.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.886' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.977199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Y—3v¢(‚(:ë £¥éÔ«¢Î® network.recv‚ 3.505952e+06) disk./.usagePercent‚78.73895010040411 disk.sda.out‚ 649.765625 disk.sda2.in‚0( proc.memory.availableMB‚ 1119.0234375 network.sent‚ 2.5523773e+07 disk.sda3.out‚ 649.765625 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.50390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚33120.55078125% disk./.usageGB‚1173.2672576904297 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0" disk.nvme0n1.in‚33120.55078125" disk.nvme0n1.out‚ 254140.734375 cpu‚25.308564046733892 proc.cpu.threads‚17 disk.dm-1.out‚ 649.765625$ disk.nvme0c0n1.out‚ 254140.734375 disk.dm-0.in‚33120.55078125 disk.dm-0.out‚ 254140.734375$ memory_percent‚93.17002296447754# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996¡q¨4£(/ _wandbruntime‚10705  _runtime‚10705ÙٰȤ( & lr-AdamW/pg2‚7.002239457062669e-05 trainer/global_step‚9999 % _timestamp‚1.7884986058964758e+09 _runtime‚10705.537008711 _step‚820 & lr-AdamW/pg1‚7.002239457062669e-05Ø& ´Í¥(Ç % _timestamp‚1.7884986058964758e+09 _runtime‚10705.537008711 _step‚820 & lr-AdamW/pg1‚7.002239457062669e-05 & lr-AdamW/pg2‚7.002239457062669e-05 trainer/global_step‚9999´ü,Vü4¦(/ _wandbruntime‚10705  _runtime‚10705w1šº(§(P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175286958 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚777178897776640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10722088883712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55736512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚787900986660352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚692247406513298 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10285215823360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10722274637312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221991489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710220829729792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884986058977113e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710220829729792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175286960 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175286960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚699936386551296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55735981 trainer/global_step‚9999 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221991489 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100508335 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚699936404554240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚710221620377600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚777178921716736 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚710221620377600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397278449 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚787901196354048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221968473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚699936386551296 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10284443178496Ýd« ¨(†P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚710221620377600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397278449 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚787901196354048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221968473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚699936386551296 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10284443178496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175286958 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚777178897776640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10722088883712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55736512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚787900986660352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚692247406513298 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10285215823360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10722274637312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221991489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 OEDeviceStatsMonitor.on_train_îêabatch_start/allocation.small_pool.current‚23016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710220829729792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884986058977113e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710220829729792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175286960 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175286960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚699936386551296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55735981 trainer/global_step‚9999 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221991489 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100508335 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚699936404554240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚710221620377600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚777178921716736 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156259885 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55741340 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚777254246119424 _runtime‚10706.430512747 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710290325075968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚700004104493568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175303832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚562 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692315261776916 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222013226 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 % _timestamp‚1.7884986067897882e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 397317060 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55741900 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397317060 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156259323 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10248421400360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 397294016 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚700004086490624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175303834 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Rý^u'«(N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚682066069727936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221990184 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚700004086490624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710289534414848 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 397294016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221990184 trainer/global_step‚9999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚700004104493568 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10286220582400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397317060 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156259323 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10248421400360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 397294016 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚700004086490624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175303834 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100517983 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 _step‚822 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚787977607753216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175303834 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10723151953408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10285447924224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156259885 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55741340 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚777254246119424 _runtime‚10706.430512747 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710290325075968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚700004104493568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175303832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚562 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692315261776916 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222013226 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 % _timestamp‚1.7884986067897882e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 397317060 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55741900 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B’¯(‚(:‡ ²¥éÔ†¹¢¯ gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.126' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…aÌr°(‚(:ç ²¥éÔÿ¹Ò± network.sent‚ 2.5528806e+07 network.recv‚ 3.509553e+06) disk./.usagePercent‚78.73892936379383 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17! disk.nvme0n1.in‚ 33294.9140625 disk.dm-0.out‚ 254686.4375 disk.sda3.out‚ 650.13671875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.50390625 disk.sda.in‚0 disk.sda1.in‚0% disk./.usageGB‚1173.2669486999512 disk.sda3.in‚0 disk.dm-1.out‚ 650.13671875 disk.sda.out‚ 650.13671875 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 33294.9140625" disk.nvme0c0n1.out‚ 254686.4375$ memory_percent‚93.17021369934082 disk.nvme0n1.out‚ 254686.4375 disk.dm-0.in‚ 33294.9140625( proc.memory.availableMB‚ 1118.9921875 cpu‚25.30823623409468Qžäóˆ±(‚(:ý Á¥éÔ„ÿ•®$ memory_percent‚93.17023754119873 proc.cpu.threads‚17 network.recv‚ 3.516101e+06) disk./.usagePercent‚78.73892961980137% disk./.usageGB‚1173.2669525146484 disk.dm-1.out‚ 650.87109375 disk.sda1.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 33294.9140625) proc.memory.availableMB‚ 1118.98828125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.5078125 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 33294.9140625# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 disk.sda3.out‚ 650.87109375 disk.drbd1260.in‚0 disk.sda.in‚0# disk.nvme0c0n1.in‚ 33294.9140625) disk.nvme0c0n1.out‚254782.11572265625 cpu‚25.307940850960296 network.sent‚ 2.5637927e+07 disk.sda.out‚ 650.87109375' disk.nvme0n1.out‚254782.11572265625$ disk.dm-0.out‚254782.11572265625¨  Ør²(‚(:‡ Á¥éÔ¼·°· gpu.0.gpu‚32.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.831' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# g1ê pu.0.uncorrectedMemoryErrors‚0 ÚF³(‚(:’ Ð¥éÔ‡ñϯ gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚132.956' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.591199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú7__ƒ´(‚(:ø Ð¥éÔÁ¦Ê´ cpu‚25.30766778190225 network.sent‚ 2.5649155e+07$ disk./.usageGB‚1173.248649597168 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 33294.9140625 disk.drbd1260.out‚ 1.5078125 disk.sda1.in‚0 network.recv‚ 3.527257e+06) disk./.usagePercent‚78.73770129565209 disk.sda2.in‚0) disk.nvme0c0n1.out‚254882.81884765625! disk.nvme0n1.in‚ 33294.9140625' disk.nvme0n1.out‚254882.81884765625 disk.sda3.in‚0 disk.dm-1.out‚ 651.4296875$ disk.dm-0.out‚254882.81884765625 disk.dm-1.in‚0 disk.sda.out‚ 651.4296875$ memory_percent‚93.17038059234619) proc.memory.availableMB‚ 1118.96484375# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17 disk.sda3.out‚ 651.4296875 disk.dm-0.in‚ 33294.9140625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.out‚0‰b’µ(‚(:‡ ߥéÔ¢î­ gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„¡Ø…¶(‚(:ú ߥéÔá² network.recv‚ 3.530868e+06' disk.nvme0n1.out‚255172.46044921875 disk.dm-0.in‚ 33294.9140625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚255172.46044921875! disk.nvme0n1.in‚ 33294.9140625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 651.68359375 disk.sda2.in‚0& proc.memory.availableMB‚ 1118.65625# proc.memory.rssMB‚ 2949.65234375 cpu‚25.307415808497222 disk.dm-1.out‚ 651.68359375 disk.drbd1260.out‚ 1.51171875) disk./.usagePercent‚78.72651146233798% disk./.usageGB‚1173.0819129943848$ disk.dm-0.out‚255172.46044921875 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 33294.9140625# memory_percent‚93.1722640991211 network.sent‚ 2.5654379e+07 disk.sda3.out‚ 651.68359375* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17Ýb4·(/ _wandbruntime‚10755  _runtime‚10755±¦ÂÕɸ(à & lr-AdamW/pg1‚6.681007022789337e-05 & lr-AdamW/pg2‚6.681007022789337e-05 trainer/global_step‚10049 % _timestamp‚1.7884986562799122e+09 _runtime‚10755.920659069 _step‚824x©x'ι(È & lr-AdamW/pg1‚6.681007022789337e-05 & lr-AdamW/pg2‚6.681007022789337e-05 trainer/global_step‚10049 % _timestamp‚1.7884986562799122e+09 _runtime‚10755.920659069 _step‚824¸‘o¦4º(/ _wandbruntime‚10755  _runtime‚10755Ü©‚(»(P QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223057375 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56002560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚695600798886406 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 176130658 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223080391 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚703321401520640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚780946314915840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚685303325080176 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚713656078396416 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10297490845590 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 176130660 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 _runtime‚10755.921677428 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 223057375 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10298261494002 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚703321383517696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚703321401520640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 399211051 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10334694878720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10774421488640 _step‚825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156993295 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚695601586574178 $ _timestamp‚1.788498656281225e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10049 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10335467523584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223080391 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156993842 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚791720736404480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚703321383517696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 399188033 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚713656869044224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚780946338855936 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 176130658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚713656078396416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚713656869044224 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚791720969166848 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10335467523584 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚685303308040816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208822272 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚713656869044224 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚791720969166848 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10335467523584 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚685303308040816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208822272 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚713656078396416 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10297490845590 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 176130660 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 _runtime‚10755.921677428 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 223057375 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10298261494002 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚703321383517696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚703321401520640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 399211051 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10334694878720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10774421488640 _step‚825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156993295 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚695601586574178 $ _timestamp‚1.788498656281225e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10049 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10335467523584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223080391 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156993842 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚791720736404480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚703321383517696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584¹òl;_4½(/ _wandbruntime‚10756  _runtime‚10756vć4#¾(ŒN OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚685369374476576 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 399249662 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚781021687198720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚695668654150024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚781021663258624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223102128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10336472282624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 399226618 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10336472282624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚703389101459968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚791797119209472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176147532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚791797347764224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚685369391515936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223079086 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 176147532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚703389083457024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚713724783081472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10335699624448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 399226618 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176147534 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223102128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚703389101459968 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176147534 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157008088 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101000385 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56008252 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10775455950848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚713725573742592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10298491985468 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 399249662 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚713725573742592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚703389083457024 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10775660565504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚10049 S?DeviceStatsMÃÀS onitor.on_train_batch_end/requested_bytes.all.freed‚695667866462044 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 % _timestamp‚1.7884986571693463e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚685369374476576 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 399249662 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚781021687198720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚695668654150024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚781021663258624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223102128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10336472282624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 399226618 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10336472282624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚703389101459968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚791797119209472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176147532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚791797347764224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚685369391515936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223079086 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 176147532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚703389083457024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚713724783081472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10335699624448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 399226618 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176147534 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223102128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚703389101459968 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176147534 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157008088 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101000385 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56008252 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10775455950848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚713725573742592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10298491985468 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 399249662 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚713725573742592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚703389083457024 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10775660565504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚10049 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚695667866462044 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 % _timestamp‚1.7884986571693463e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚717091327063040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 % _timestamp‚1.7884987033558679e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚717091327063040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224169293 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚10802.996589007 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157740459 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚706706380484096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56266782 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚706706398487040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚706706380484096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157739917 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10347561430698 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 176974360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚784713732055040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224146277 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10384946578944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚706706398487040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚688606629828816 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10385719223808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 trainer/global_step‚10099 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224146277 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10826978096640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚829 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚717092117710848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2;$Æ3Î(‡P TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10384946578944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚706706380484096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56266782 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚706706398487040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚706706380484096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157739917 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10347561430698 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 176974360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚784713732055040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224146277 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10384946578944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚706706398487040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚688606629828816 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10385719223808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 trainer/globalg;+Ú_step‚10099 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224146277 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10826978096640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚829 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚717092117710848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 401120635 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56267322 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚795540516009472 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 176974360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚698954978947286 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚717091327063040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 % _timestamp‚1.7884987033558679e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚717091327063040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224169293 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚10802.996589007 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157740459 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚784789104337920 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 401182264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚546 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10828026669568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚706774080423424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56272700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 _runtime‚10804.190343169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224191030 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101482785 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176991234 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 401182264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10827828346368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10349333219196 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 _step‚830 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚795617131007488 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10386723982848 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚699022046523132 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10385951324672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚688672713303936 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101482783 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157754937 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚706774098426368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10386723982848 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚717160031748096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚706774098426368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56272154¼MI5'Ñ(‘N UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚784789104337920 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 401182264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚546 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10828026669568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚706774080423424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56272700 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 _runtime‚10804.190343169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224191030 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101482785 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176991234 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 401182264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10827828346368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10349333219196 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 _step‚830 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚795617131007488 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10386723982848 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚699022046523132 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10385951324672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚688672713303936 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101482783 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157754937 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚706774098426368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10386723982848 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚717160031748096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚706774098426368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56272154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚699021258835152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚706774080423424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10348562570576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 401159220 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚717160822409216 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚717160822409216 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚795616908744192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚784789080397824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚548 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0¾nÄž4Ò(/ _wandbruntime‚10804  _runtime‚10804fu—êZÓ(Ô /loss/FRt2/assignment_loss‚3.244100837243928 0loss/FBt2/assignment_loss‚0.2651516810045161 /loss/FRt3/detection_loss‚2.7599907362902605 _runtime‚10804.19235479 /loss/FBt1/assignment_loss‚0.306246306021301 .loss/FRt2/detection_loss‚2.338390407930624 0loss/SRqqt1/detection_loss‚2.158201170690132 1loss/SRqqt4/detection_loss‚0.9248839791080999 % _timestamp‚1.7884987045520463e+09 /loss/FRt1/assignment_loss‚3.471007309680761 /loss/FBt1/detection_loss‚2.0093084363376392 0loss/FBt4/detection_loss‚0.40938238558077883 0loss/FRt3/assignment_loss‚2.2646026920389244 2loss/SRqqt1/assignment_loss‚1.1842556952527075 1loss/FBt3/assignment_loss‚0.24644667135985898 1loss/SRqqt2/detection_loss‚2.3446059584617616 1loss/SRqqt3/detection_loss‚1.9440053407812403 epoch‚8 2loss/SRqqt2/assignment_loss‚1.4313604133824507 1loss/SRqqt3/assignment_loss‚1.434853765237633 .loss/FRt1/detection_loss‚2.604881277697724 /loss/FBt2/detection_loss‚2.3268227495930414 .loss/FBt3/detection_loss‚1.979194530418941 %loss/total_loss‚39.15448052916741 trainer/global_step‚10099 0loss/FRt4/assignment_loss‚1.1247350565261311 2loss/SRqqt4/assignment_loss‚0.9539995476952754 1loss/FBt4/assignment_loss‚0.19639054826944394 .loss/FRt4/detection_loss‚1.231663032564231 _step‚831y·)È_Ô(Ù % _timestamp‚1.7884987045520463e+09 /loss/FRt1/assignment_loss‚3.471007309680761 /loss/FBt1/detection_loss‚2.0093084363376392 0loss/FBt4/detection_loss‚0.40938238558077883 0loss/FRt3/assignment_loss‚2.2646026920389244 2loss/SRqqt1/assignment_loss‚1.1842556952527075 1loss/FBt3/assignment_loss‚0.24644667135985898 1loss/SRqqt2/detection_loss‚2.3446059584617616 1loss/SRqqt3/detection_loss‚1.9440053407812403 epoch‚8 2loss/SRqqt2/assignment_loss‚1.4313604133824507 1loss/SRqqt3/assignment_loss‚1.434853765237633 .loss/FRt1/detection_loss‚2.604881277697724 /loss/FBt2/detection_loss‚2.3268227495930414 .loss/FBt3/detection_loss‚1.979194530418941 %loss/total_loss‚39.15448052916741 trainer/global_step‚10099 0loss/FRt4/assignment_loss‚1.1247350565261311 2loss/SRqqt4/assignment_loss‚0.9539995476952754 1loss/FBt4/assignment_loss‚0.19639054826944394 .loss/FRt4/detection_loss‚1.231663032564231 _step‚831 /loss/FRt2/assignment_loss‚3.244100837243928 0loss/FBt2/assignment_loss‚0.2651516810045161 /loss/FRt3/detection_loss‚2.7599907362902605 _runtime‚10804.19235479 /loss/FBt1/assignment_loss‚0.306246306021301 .loss/FRt2/detection_loss‚2.338390407930624 0loss/SRqqt1/detection_loss‚2.158201170690132 1loss/SRqqt4/detection_loss‚0.9248839791080999¿’ìy‘Õ(‚(:† ›¦éÔˆ‹È° gpu.0.gpu‚33.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.465' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.293 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³•ª‹Ö(‚(:€ ›¦éÔ•¹ø²' disk.nvme0n1.out‚256592.95849609375 disk.drbd1260.out‚ 1.51953125 disk.sda.in‚0$ memory_percent‚93.17567348480225) proc.memory.availableMB‚ 1118.09765625# proc.memory.rssMB‚ 2949.65234375% disk./.usageGB‚1173.5779571533203 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 653.84765625* proc.memory.percent‚18.003249168395996 cpu‚25.30622323120643 proc.cpu.threads‚17 network.sent‚ 2.5777455e+07 network.recv‚ 3.549453e+06$ disk.dm-0.out‚256592.95849609375 disk.dm-1.in‚0) disk./.usagePercent‚78.75980140205291 disk.sda2.in‚0" disk.nvme0n1.in‚33427.23046875 disk.dm-0.in‚33427.23046875 disk.sda3.out‚ 653.84765625 disk.dm-1.out‚ 653.84765625 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚33427.23046875) disk.nvme0c0n1.out‚256592.95849609375†þº\’×(‚(:‡ ª¦éÔÑÐß­ gpu.0.gpu‚33.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.769' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.7538 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓTÙˆØ(‚(:ý ª¦éÔ³Éñ* proc.memory.percent‚18.003249168395996 network.sent‚ 2.588527e+07 network.recv‚ 3.554991e+06 disk.sda1.in‚0$ disk.nvme0c0n1.in‚33594.87109375$ disk.dm-0.out‚256944.92236328125 disk.sda3.out‚ 654.4453125 disk.drbd1260.out‚ 1.51953125$ memory_percent‚93.17262172698975" disk.nvme0n1.in‚33594.87109375 disk.sda3.in‚0 cpu‚25.305969422775465 proc.cpu.threads‚17 disk.sda.out‚ 654.4453125 disk.sda2.in‚0) disk.nvme0c0n1.out‚256944.92236328125' disk.nvme0n1.out‚256944.92236328125 disk.dm-0.in‚33594.87109375 disk.dm-1.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2949.65234375) disk./.usagePercent‚78.77657911181689% disk./.usageGB‚1173.8279571533203 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 654.4453125) proc.memory.availableMB‚ 1118.59765625ŸŽ²=’Ù(‚(:‡ ¹¦éÔÄ¡Š° gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|aži¹Ú(‚(:ã ¹¦éÔ¸²ç° network.recv‚ 3.559899e+06 disk.drbd1260.in‚0 disk.dm-0.in‚33594.87109375 disk.dm-0.out‚ 257188.75 disk.dm-1.in‚0 disk.dm-1.out‚ 654.90234375$ memory_percent‚93.17412376403809( proc.memory.availableMB‚ 1118.3515625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚33594.87109375 disk.nvme0n1.out‚ 257188.75 disk.sda3.in‚0# proc.memory.rssMB‚ 2949.65234375 cpu‚25.305690056594685( disk./.usagePercent‚78.7767915980703% disk./.usageGB‚1173.8311233520508 disk.drbd1260.out‚ 1.5234375 disk.sda.in‚0 disk.sda1.in‚0" disk.nvme0n1.in‚33594.87109375 disk.sda3.outƒ‚µ‚ 654.90234375 proc.cpu.threads‚17 network.sent‚ 2.5891587e+07 disk.sda.out‚ 654.90234375 disk.nvme0c0n1.out‚ 257188.75* proc.memory.percent‚18.003249168395996­]m4Û(/ _wandbruntime‚10855  _runtime‚10855ž*Þ¢ËÜ(Å ' lr-AdamW/pg2‚6.0591960430434553e-05 trainer/global_step‚10149 % _timestamp‚1.7884987554272406e+09 _runtime‚10855.067533027 _step‚832 ' lr-AdamW/pg1‚6.0591960430434553e-05„®=BÐÝ(Ê trainer/global_step‚10149 % _timestamp‚1.7884987554272406e+09 _runtime‚10855.067533027 _step‚832 ' lr-AdamW/pg1‚6.0591960430434553e-05 ' lr-AdamW/pg2‚6.0591960430434553e-05Àð5z94Þ(/ _wandbruntime‚10855  _runtime‚10855z5‡(ß(‚P YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚720527366377472 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10397632015806 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225235179 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101955537 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚788481149194240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚691909951616816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚10149 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10879221102592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚833 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚710091395453440 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 177818060 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10435198279168 _runtime‚10855.068679948 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225258195 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚720526575729664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚799360370296832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚710091377450496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 403076255 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 177818060 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56534073 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚702308371320394 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚799360592573440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10879419439104 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10435198279168 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720526575729664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225258195 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884987554281945e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 158489610 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10435970924032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 177818058 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 225235179 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56533539 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚710091377450496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101955535 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328¼ì|Ý (à(‡P XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101955535 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚720527366377472 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10397632015806 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225235179 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101955537 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚788481149194240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚691909951616816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚10149 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10879221102592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚833 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚710091395453440 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 177818060 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10435198279168 _runtime‚10855.068679948 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225258195 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚720526575729664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚799360370296832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚710091377450496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 403076255 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 177818060 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56534073 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚702308371320394 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚799360592573440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10879419439104 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10435198279168 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720526575729664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225258195 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884987554281945e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 158489610 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10435970924032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 177818058 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 225235179 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56533539 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚710091377450496Ádû4á(/ _wandbruntime‚10855  _runtime‚10855ôãéÞ'â(‹N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 177834934 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚710159095392768 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚710159077389824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 403091822 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚702374651208260 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 403114866 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 _step‚834 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225279932 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 403114866 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚799436783995904 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚788556497537024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚799436997870592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚702375438896240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _runtime‚10855.939035315 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225256890 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101965185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10880286458880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚788556521477120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 403091822 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56538812 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10880476393472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚691976035091936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚720596071075840 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225256890 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101965183 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56539341 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10398633155684 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10436203024896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚710159095392768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚720595280414720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225279932 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚10149 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177834932 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10436975683072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10399403804304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 $ _timestamp‚1.788498756298296e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 177834934 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 177834932 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158503995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚691976018052576 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚710159077389824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ñß»7¨ã(N VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚788556521477120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 403091822 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56538812 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10880476393472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚691976035091936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚7205960\ÖÇn!71075840 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225256890 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101965183 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56539341 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10398633155684 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10436203024896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚710159095392768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚720595280414720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225279932 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚10149 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177834932 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10436975683072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10399403804304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 $ _timestamp‚1.788498756298296e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 177834934 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 177834932 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158503995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚691976018052576 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚710159077389824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 177834934 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚710159095392768 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚710159077389824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 403091822 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚702374651208260 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 403114866 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 _step‚834 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225279932 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 403114866 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚799436783995904 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚788556497537024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚799436997870592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚702375438896240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _runtime‚10855.939035315 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225256890 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101965185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10880286458880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2ÂGU(¯4ä(/ _wandbruntime‚10855  _runtime‚10855áqUÿ_å(Ù 2loss/SRqqt1/assignment_loss‚1.5090718450290816 2loss/SRqqt3/assignment_loss‚1.4729946319691158 /loss/FBt1/detection_loss‚3.2959537506103516 %loss/total_loss‚39.81231921250843 2loss/SRqqt2/assignment_loss‚1.3399239511866319 2loss/FBt3/assignment_loss‚0.055393605697767204 /loss/FBt3/detection_loss‚2.7566883734294345 trainer/global_step‚10149 0loss/FRt1/assignment_loss‚2.9694488687174663 0loss/FRt4/assignment_loss‚0.9559623328844706 1loss/FBt2/assignment_loss‚0.17168948231523398 .loss/FRt1/detection_loss‚2.301844035053537 1loss/FBt1/assignment_loss‚0.16441907725371982 1loss/SRqqt2/detection_loss‚2.5226975522543253 /loss/FBt4/detection_loss‚0.6288859596068506 0loss/FRt3/assignment_loss‚2.6567104146594094 1loss/SRqqt4/assignment_loss‚1.117160393825073 .loss/FRt3/detection_loss‚2.398565935237067 $ _timestamp‚1.788498756300459e+09 /loss/FBt2/detection_loss‚1.9745180984338124 epoch‚8 _step‚835 /loss/FRt2/assignment_loss‚3.048798441886902 /loss/FRt2/detection_loss‚2.2720918280554088 /loss/SRqqt1/detection_loss‚2.41153029778174 1loss/SRqqt4/detection_loss‚0.7424048459513134 2loss/FBt4/assignment_loss‚0.055657900202793084 /loss/FRt4/detection_loss‚1.3907340050737063 0loss/SRqqt3/detection_loss‚1.599173585393212 _runtime‚10855.940625662¦çdæ(Þ 1loss/SRqqt4/assignment_loss‚1.117160393825073 .loss/FRt3/detection_loss‚2.398565935237067 $ _timestamp‚1.788498756300459e+09 /loss/FBt2/detection_loss‚1.9745180984338124 epoch‚8 _step‚835 /loss/FRt2/assignment_loss‚3.048798441886902 /loss/FRt2/detection_loss‚2.2720918280554088 /loss/SRqqt1/detection_loss‚2.41153029778174 1loss/SRqqt4/detection_loss‚0.7424048459513134 2loss/FBt4/assignment_loss‚0.055657900202793084 /loss/FRt4/detection_loss‚1.3907340050737063 0loss/SRqqt3/detection_loss‚1.599173585393212 _runtime‚10855.940625662 2loss/SRqqt1/assignment_loss‚1.5090718450290816 2loss/SRqqt3/assignment_loss‚1.4729946319691158 /loss/FBt1/detection_loss‚3.2959537506103516 %loss/total_loss‚39.81231921250843 2loss/SRqqt2/assignment_loss‚1.3399239511866319 2loss/FBt3/assignment_loss‚0.055393605697767204 /loss/FBt3/detection_loss‚2.7566883734294345 trainer/global_step‚10149 0loss/FRt1/assignment_loss‚2.9694488687174663 0loss/FRt4/assignment_loss‚0.9559623328844706 1loss/FBt2/assignment_loss‚0.17168948231523398 .loss/FRt1/detection_loss‚2.301844035053537 1loss/FBt1/assignment_loss‚0.16441907725371982 1loss/SRqqt2/detection_loss‚2.5226975522543253 /loss/FBt4/detection_loss‚0.6288859596068506 0loss/FRt3/assignment_loss‚2.6567104146594094Ã3_>’ç(‚(:‡ ȦéÔ¨Êý° gpu.0.gpu‚19.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%çTœ{è(‚(:ð ȦéÔ¸žò±# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17% disk./.usageGB‚1173.8625106811523 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 655.44921875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.5234375 disk.sda.in‚0$ disk.nvme0c0n1.in‚33594.87109375) disk./.usagePercent‚78.77889802806203 disk.sda1.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.out‚ 257524.203125" disk.nvme0n1.in‚33594.87109375" disk.nvme0n1.out‚ 257524.203125 disk.dm-0.in‚33594.87109375 disk.dm-0.out‚ 257524.203125 network.sent‚ 2.590958e+07 disk.sda.out‚ 655.44921875) proc.memory.availableMB‚ 1118.55859375 cpu‚25.305402729579342 network.recv‚ 3.563553e+06 disk.sda3.out‚ 655.44921875 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚93.17286014556885Þ'w¾é(‚(:… צéÔ쬸± gpu.0.gpu‚51.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.39' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.278 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pËvê(‚(:ë צéÔñ¹²¸ disk.sda1.out‚0" disk.nvme0n1.in‚33594.87109375 disk.dm-0.in‚33594.87109375# disk.dm-0.out‚258044.0595703125$ memory_percent‚93.17450523376465 network.recv‚ 3.569471e+06) disk./.usagePercent‚78.77889802806203 disk.dm-1.out‚655.75 disk.sda.out‚655.75( disk.nvme0c0n1.out‚258044.0595703125( proc.memory.availableMB‚ 1118.2890625 proc.cpu.threads‚17 network.sent‚ 2.6004867e+07 disk.sda1.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚33594.87109375# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 cpu‚25.305113654597555 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.52734375 disk.sda2.in‚0& disk.nvme0n1.out‚258044.0595703125% disk./.usageGB‚1173.8625106811523 disk.sda3.out‚655.75 disk.sda.in‚0ÈYÄ ë(‚(:‚ æ¦éÔü¿à® gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.9' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.38 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢.'Ɔì(‚(:û æ¦éÔÏ„°# disk.dm-0.out‚258203.0712890625# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 cpu‚25.304849978489745 network.recv‚ 3.579897e+06 disk.dm-0.in‚33745.15234375) proc.memory.availableMB‚ 1118.53515625 proc.cpu.threads‚17% disk./.usageGB‚1173.8625106811523 disk.dm-1.out‚ 656.1328125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.52734375 disk.sda.in‚0 disk.sda.out‚ 656.1328125 disk.sda1.out‚0" disk.nvme0n1.in‚33745.12109375 disk.sda1.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚33745.06640625( disk.nvme0c0n1.out‚258203.0712890625& disk.nvme0n1.out‚258203.0712890625 disk.sda3.in‚0 disk.sda3.out‚ 656.1328125$ memory_percent‚93.17300319671631 network.sent‚ 2.6013165e+07) disk./.usagePercent‚78.77889802806203 disk.dm-1.in‚0 disk.sda2.in‚0¸øüf’í(‚(:‡ õ¦éÔúãÙ¯ gpu.0.gpu‚34.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]‰‚î(‚(:÷ õ¦éÔ…ø‡±) disk./.usagePercent‚78.82083974239661% disk./.usageGB‚1174.4874725341797$ disk.nvme0c0n1.in‚33753.72265625) disk.nvme0c0n1.out‚258459.74560546875' disk.nvme0n1.out‚258459.74560546875$ disk.dm-0.out‚258459.74560546875 disk.drbd1260.in‚0( proc.memory.availableMB‚ 1118.7890625 network.sent‚ 2.6022478e+07" disk.nvme0n1.in‚33753.72265625 disk.dm-0.in‚33753.72265625 disk.sda3.out‚ 656.609375 disk.sda.in‚0 disk.sda.out‚ 656.609375 disk.sda2.out‚0* proc.memory.percent‚18.003249168395996 disk.dm-1.out‚ 656.609375 network.recv‚ 3.585556e+06 disk.dm-1.in‚0 disk.drbd1260.out‚1.53125 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚93.17145347595215# proc.memory.rssMB‚ 2949.65234375 disk.sda3.in‚0 disk.sda1.out‚0 cpu‚25.304588186888235 proc.cpu.threads‚17Ga4ï(/ _wandbruntime‚10905  _runtime‚10905Ö^™7Éð(à & lr-AdamW/pg2‚5.758747082047076e-05 trainer/global_step‚10199 % _timestamp‚1.7884988056878915e+09 _runtime‚10905.328251233 _step‚836 & lr-AdamW/pg1‚5.758747082047076e-05¨Z“ÀÎñ(È & lr-AdamW/pg2‚5.758747082047076e-05 trainer/global_step‚10199 % _timestamp‚1.7884988056878915e+09 _runtime‚10905.328251233 _step‚836 & lr-AdamW/pg1‚5.758747082047076e-05Äsô†Ç4ò(/ _wandbruntime‚10905  _runtime‚10905»—·B(ó(‚P LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚723961824396288 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10486222624256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚723962615044096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚713476392419840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10485449979392 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 178661760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚695213290444176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚713476374416896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚723961824396288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚713476374416896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10486222624256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10448473249326 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10931705921536 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102437937 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102437935 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159238600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚705660976005730 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226324081 _runtime‚10905.329484627 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 178661760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10447702600914 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 178661758 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56800663 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10931906355200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚10199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226324081 _runtime‚10905.329484627 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 178661760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10447702600914 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 178661758 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56800663 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10931906355200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 trainer/global_step‚10199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚723961824396288Å÷º>4õ(/ _wandbruntime‚10906  _runtime‚10906Ïœbã'ö(ŒN B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10932953052160 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 _runtime‚10906.301070938 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 trainer/global_step‚10199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚713544074356224 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226345792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 405024424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56805881 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10449474389412 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚724030529081344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10487227383296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚803256891668480 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚713544092359168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226345792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚724031319742464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 405047468 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159252935 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 226368834 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159253466 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚713544074356224 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102447585 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚803256891668480 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚713544092359168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226345792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚724031319742464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 405047468 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159252935 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 226368834 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159253466 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚713544074356224 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102447585 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 _runtime‚10906.301070938 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 trainer/global_step‚10199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚713544074356224 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226345792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ÆEß4ø(/ _wandbruntime‚10906  _runtime‚10906XNÔ&Zù(Ô .loss/FBt1/detection_loss‚3.638346532980601 epoch‚8 1loss/FBt1/assignment_loss‚0.17646750334339836 1loss/FBt4/assignment_loss‚0.13647297260940422 trainer/global_step‚10199 % _timestamp‚1.7884988066636631e+09 0loss/FRt3/assignment_loss‚2.2962272552882923 1loss/FBt3/assignment_loss‚0.45113318616421905 -loss/FRt3/detection_loss‚2.40156602903324 2loss/SRqqt4/assignment_loss‚0.9192714851266629 1loss/SRqqt4/detection_loss‚0.8934236448686356 /loss/FRt4/assignment_loss‚1.235839285348591 1loss/SRqqt1/assignment_loss‚1.726793224577393 1loss/FBt2/assignment_loss‚0.16465823348287884 0loss/SRqqt3/detection_loss‚1.759039647703947 .loss/FBt3/detection_loss‚2.342556683222453 0loss/FRt1/assignment_loss‚3.4331016855107412 .loss/FRt2/assignment_loss‚3.26602344359121 _runtime‚10906.30403319 2loss/SRqqt3/assignment_loss‚1.0400109187981417 /loss/FRt4/detection_loss‚1.4320137261805173 0loss/SRqqt2/detection_loss‚2.159288889938785 /loss/FBt2/detection_loss‚2.3038652539253235 &loss/total_loss‚40.687670669729364 _step‚839 0loss/SRqqt1/detection_loss‚2.087329242910658 0loss/FBt4/detection_loss‚0.42533556845062126 /loss/FRt2/detection_loss‚2.3344267070713083 1loss/SRqqt2/assignment_loss‚1.413577471529284 /loss/FRt1/detection_loss‚2.6509020780730577ù¶y›_ú(Ù 2loss/SRqqt3/assignment_loss‚1.0400109187981417 /loss/FRt4/detection_loss‚1.4320137261805173 0loss/SRqqt2/detection_loss‚2.159288889938785 /loss/FBt2/detection_loss‚2.3038652539253235 &loss/total_loss‚40.687670669729364 _step‚839 0loss/SRqqt1/detection_loss‚2.087329242910658 0loss/FBt4/detection_loss‚0.42533556845062126 /loss/FRt2/detection_loss‚2.3344267070713083 1loss/SRqqt2/assignment_loss‚1.413577471529284 /loss/FRt1/detection_loss‚2.6509020780730577 .loss/FBt1/detection_loss‚3.638346532980601 epoch‚8 1loss/FBt1/assignment_loss‚0.17646750334339836 1loss/FBt4/assignment_loss‚0.13647297260940422 trainer/global_step‚10199 % _timestamp‚1.7884988066636631e+09 0loss/FRt3/assignment_loss‚2.2962272552882923 1loss/FBt3/assignment_loss‚0.45113318616421905 -loss/FRt3/detection_loss‚2.40156602903324 2loss/SRqqt4/assignment_loss‚0.9192714851266629 1loss/SRqqt4/detection_loss‚0.8934236448686356 /loss/FRt4/assignment_loss‚1.235839285348591 1loss/SRqqt1/assignment_loss‚1.726793224577393 1loss/FBt2/assignment_loss‚0.16465823348287884 0loss/SRqqt3/detection_loss‚1.759039647703947 .loss/FBt3/detection_loss‚2.342556683222453 0loss/FRt1/assignment_loss‚3.4331016855107412 .loss/FRt2/assignment_loss‚3.26602344359121 _runtime‚10906.30403319Ç`ƒ%’û(‚(:‡ „§éÔŠÔô³ gpu.0.gpu‚23.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.161' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.6322 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"SK{„ü(‚(:ù „§éԢ짵 disk.sda.in‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 1118.265625 proc.cpu.threads‚17' disk.nvme0n1.out‚258811.20263671875 disk.sda3.out‚ 657.1015625 disk.sda.out‚ 657.1015625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 657.1015625) disk./.usagePercent‚78.82926315829573% disk./.usageGB‚1174.6129875183105$ disk.nvme0c0n1.in‚33753.72265625" disk.nvme0n1.in‚33753.72265625 disk.dm-0.in‚33753.72265625$ disk.dm-0.out‚258811.20263671875 network.recv‚ 3.590154e+06 disk.drbd1260.out‚1.53125 disk.sda2.out‚0$ memory_percent‚93.17464828491211# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 cpu‚25.304311836281908 disk.dm-1.in‚0 network.sent‚ 2.6026522e+07) disk.nvme0c0n1.out‚258811.20263671875 disk.drbd1260.in‚0\Q¡ý(‚(:’ “§éÔ®ØÄ® gpu.0.gpu‚37.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.373' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.674599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dù¥ß|þ(‚(:ñ “§éÔ÷Ìú±" disk.nvme0n1.in‚33753.72265625$ disk.nvme0n1.out‚259172.13671875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd1260.out‚ 1.53515625 network.sent‚ 2.6136097e+07% disk./.usageGB‚1174.7448043823242 disk.dm-0.in‚33753.72265625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 657.359375 disk.sda1.out‚0 network.recv‚ 3.596237e+06 disk.sda2.out‚0$ disk.nvme0c0n1.in‚33753.72265625$ memory_percent‚93.17467212677002) proc.memory.availableMB‚ 1118.26171875# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996( disk./.usagePercent‚78.8381094986414! disk.dm-0.out‚259172.13671875 disk.sda3.out‚ 657.359375 disk.dm-1.out‚ 657.359375 disk.sda2.in‚0& disk.nvme0c0n1.out‚259172.13671875 cpu‚25.304049319394345¤§J’ÿ(‚(:‡ ¢§éÔ¨˜à­ gpu.0.gpu‚33.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.613' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯[Z~€)‚(:ó ¢§éÔ§Éà°% disk.nvme0c0n1.out‚259596.6171875" disk.nvme0n1.in‚33753.72265625$ memory_percent‚93.17469596862793 network.sent‚ 2.6139259e+07% disk./.usageGB‚1174.7907829284668 disk.dm-0.out‚259596.6171875 disk.sda3.in‚0# disk.nvme0n1.out‚259596.6171875# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 cpu‚25.303777458161424 disk.drbd1260.out‚ 1.53515625 disk.sda.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1118.2578125 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.sda.out‚ 657.76953125( disk./.usagePercent‚78.8411951574516 disk.sda3.out‚ 657.76953125 disk.dm-1.in‚0 disk.dm-1.out‚ 657.76953125 disk.sda1.out‚0 disk.sda2.in‚0 network.recv‚ 3.599845e+06 disk.dm-0.in‚33753.72265625 disk.sda1.in‚0$ disk.nvme0c0n1.in‚33753.72265625K* {4)/ _wandbruntime‚10956  _runtime‚10956N³¹nÆ‚)À _runtime‚10956.847891583 _step‚840 % lr-AdamW/pg1‚5.46535311018617e-05 % lr-AdamW/pg2‚5.46535311018617e-05 trainer/global_step‚10249 $ _timestamp‚1.788498857207379e+09½PÙ\˃)Å % lr-AdamW/pg1‚5.46535311018617e-05 % lr-AdamW/pg2‚5.46535311018617e-05 trainer/global_step‚10249 $ _timestamp‚1.788498857207379e+09 _runtime‚10956.847891583 _step‚840ȇ/4„)/ _wandbruntime‚10956  _runtime‚10956Á–,ÃÏ…)‚P aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚796015983472640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚807000251125248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reseã±ù9&rved_bytes.all.peak‚ 12155092992 _step‚841 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179505458 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102920337 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185753600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚716861371383296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57067898 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 179505458 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚698516595192816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚716861389386240 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚796016007412736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159988779 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57068442 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10984267652608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 406918441 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227412983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159988233 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚716861371383296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚709015156066610 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227435999 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 406941459 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚727397073062912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚716861389386240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10536474324480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209693696 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10536474324480 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 227412983 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚698516612232176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚727397863710720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 % _timestamp‚1.7884988572091587e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10535701679616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10497773186022 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10498543834434 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚727397863710720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10535701679616 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚807000460818944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚709014368378838 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚10956.850052224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚727397073062912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680F/Qà (†)‡P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10498543834434 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚727397863710720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10535701679616 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚807000460818944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚709014368378838 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚10956.850052224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚727397073062912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚796015983472640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚807000251125248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 _step‚841 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179505458 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102920337 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 185753600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚716861371383296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57067898 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 179505458 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚698516595192816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚716861389386240 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚796016007412736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159988779 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57068442 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10984267652608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 406918441 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227412983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159988233 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚716861371383296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚709015156066610 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227435999 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 406941459 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚727397073062912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚716861389386240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10536474324480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209693696 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10536474324480 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 227412983 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚698516612232176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚727397863710720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 % _timestamp‚1.7884988572091587e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10535701679616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10497773186022ÉêºhB4‡)/ _wandbruntime‚10957  _runtime‚10957Â6µ'ˆ)ŒN cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚796091355755520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179522334 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227434694 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚10957.798879528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10985318000128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10498774325900 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚716929071322624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 406957026 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 179522334 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227457736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚807076649815552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 406957026 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227457736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 406980070 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 179522332 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10537479083520 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160003762 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚727466568409088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160003219 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10536706425344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10249 % _timestamp‚1.7884988581572962e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚796091331815424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚709082223642456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚727465777747968 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179522332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57073777 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102929985 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚807076865787392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚716929071322624 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚716929089325568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57073236 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚709081435954476 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227434694 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10499544974520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10985510031872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 406980070 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚727466568409088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚716929089325568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚698582678667936 _step‚842 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980j¨ñ½Q ‰)‘N WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚716929071322624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 406957026 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 179522334 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227457736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚807076649815552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 406957026 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227457736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 406980070 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 179522332 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10537479083520 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160003762 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚727466568409088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitE4"ŸÆor.on_train_batch_end/inactive_split.all.freed‚ 160003219 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10536706425344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10249 % _timestamp‚1.7884988581572962e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚796091331815424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚709082223642456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚727465777747968 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179522332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57073777 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102929985 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚807076865787392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚716929071322624 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚716929089325568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57073236 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215971840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚709081435954476 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227434694 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10499544974520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10985510031872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 406980070 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚727466568409088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚716929089325568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚698582678667936 _step‚842 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚796091355755520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179522334 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227434694 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚10957.798879528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10985318000128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10498774325900 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Êj ñ4Š)/ _wandbruntime‚10957  _runtime‚10957¢®Ã]‹)× /loss/FRt3/detection_loss‚2.8612603462404675 0loss/SRqqt2/detection_loss‚2.236625841435264 /loss/FBt4/detection_loss‚0.5016791062057018 0loss/FRt4/assignment_loss‚0.9481288171433783 2loss/SRqqt1/assignment_loss‚1.1557792953469537 1loss/FBt2/assignment_loss‚0.12386089359210928 &loss/total_loss‚38.268463205504105 _step‚843 1loss/FBt3/assignment_loss‚0.07566859940682728 0loss/SRqqt1/detection_loss‚2.270175175233321 _runtime‚10957.800436472 /loss/FRt3/assignment_loss‚2.341785987218221 2loss/SRqqt3/assignment_loss‚1.4472095919202785 .loss/FBt3/detection_loss‚2.677014023065567 trainer/global_step‚10249 % _timestamp‚1.7884988581601613e+09 /loss/FRt4/detection_loss‚1.4913086700555567 .loss/FBt1/detection_loss‚2.132708495313471 epoch‚8 2loss/SRqqt2/assignment_loss‚1.6226077912484898 1loss/FBt1/assignment_loss‚0.04453011284666401 1loss/SRqqt3/detection_loss‚1.8976884619617949 0loss/SRqqt4/detection_loss‚0.737415354356778 0loss/FRt1/assignment_loss‚2.8292572789413986 /loss/FRt2/assignment_loss‚2.247411733164507 1loss/FBt4/assignment_loss‚0.14858321158407997 /loss/FRt1/detection_loss‚2.2906690213915915 .loss/FRt2/detection_loss‚2.511641781748822 2loss/SRqqt4/assignment_loss‚0.9701693931223863 .loss/FBt2/detection_loss‚2.705284222960472ÑãY!bŒ)Ü _runtime‚10957.800436472 /loss/FRt3/assignment_loss‚2.341785987218221 2loss/SRqqt3/assignment_loss‚1.4472095919202785 .loss/FBt3/detection_loss‚2.677014023065567 trainer/global_step‚10249 % _timestamp‚1.7884988581601613e+09 /loss/FRt4/detection_loss‚1.4913086700555567 .loss/FBt1/detection_loss‚2.132708495313471 epoch‚8 2loss/SRqqt2/assignment_loss‚1.6226077912484898 1loss/FBt1/assignment_loss‚0.04453011284666401 1loss/SRqqt3/detection_loss‚1.8976884619617949 0loss/SRqqt4/detection_loss‚0.737415354356778 0loss/FRt1/assignment_loss‚2.8292572789413986 /loss/FRt2/assignment_loss‚2.247411733164507 1loss/FBt4/assignment_loss‚0.14858321158407997 /loss/FRt1/detection_loss‚2.2906690213915915 .loss/FRt2/detection_loss‚2.511641781748822 2loss/SRqqt4/assignment_loss‚0.9701693931223863 .loss/FBt2/detection_loss‚2.705284222960472 /loss/FRt3/detection_loss‚2.8612603462404675 0loss/SRqqt2/detection_loss‚2.236625841435264 /loss/FBt4/detection_loss‚0.5016791062057018 0loss/FRt4/assignment_loss‚0.9481288171433783 2loss/SRqqt1/assignment_loss‚1.1557792953469537 1loss/FBt2/assignment_loss‚0.12386089359210928 &loss/total_loss‚38.268463205504105 _step‚843 1loss/FBt3/assignment_loss‚0.07566859940682728 0loss/SRqqt1/detection_loss‚2.270175175233321ËUãJ‘)‚(:† ±§éÔª¶‰® gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚127.595' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.519 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03‹”¿ŒŽ)‚(: ±§éÔýÈø° disk.sda3.in‚0% disk./.usageGB‚1174.6705055236816 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 658.33984375# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 cpu‚25.303499816894224 proc.cpu.threads‚17 disk.dm-1.out‚ 658.33984375 disk.sda.in‚0$ disk.nvme0c0n1.in‚33876.79296875" disk.nvme0n1.in‚33876.79296875$ disk.dm-0.out‚260021.45361328125$ memory_percent‚93.17362308502197 network.sent‚ 2.6168691e+07) disk./.usagePercent‚78.83312323989699 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1118.43359375 network.recv‚ 3.624865e+06 disk.drbd1260.out‚ 1.54296875 disk.sda.out‚ 658.33984375 disk.sda1.in‚0) disk.nvme0c0n1.out‚260021.45361328125' disk.nvme0n1.out‚260021.45361328125 disk.dm-0.in‚33876.79296875Eª¦§’)‚(:‡ À§éÔëþ’¯ gpu.0.gpu‚25.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.969' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹à.)‚(:ô À§éÔ£¦ƒ³% disk./.usageGB‚1174.9204864501953$ disk.nvme0c0n1.in‚33876.79296875" disk.nvme0n1.in‚33876.79296875' disk.nvme0n1.out‚260373.81689453125 disk.dm-0.in‚33876.79296875 disk.sda3.in‚0 network.sent‚ 2.6264251e+07( disk./.usagePercent‚78.8498996696233) disk.nvme0c0n1.out‚260373.81689453125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.54296875 disk.sda.out‚ 658.734375 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2949.65234375 cpu‚25.3032593240115$ disk.dm-0.out‚260373.81689453125 disk.sda3.out‚ 658.734375 disk.sda1.out‚0$ memory_percent‚93.17359924316406% proc.memory.availableMB‚ 1118.4375* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17 disk.dm-1.out‚ 658.734375 network.recv‚ 3.631976e+06j¨Œ‘‘)‚(:† ϧéÔÏŒí° gpu.0.gpu‚6.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.433' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.6866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷Þ¶6†’)‚(:û ϧéÔßâü³ disk.dm-1.out‚ 659.0703125 network.recv‚ 3.635547e+06 disk.dm-0.in‚33876.79296875 disk.sda3.in‚0) disk.nvme0c0n1.out‚260533.23876953125" disk.nvme0n1.in‚33876.79296875* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚93.17514896392822) proc.memory.availableMB‚ 1118.18359375# proc.memory.rssMB‚ 2949.65234375 disk.drbd1260.in‚0 network.sent‚ 2.6269346e+07% disk./.usageGB‚1174.9204864501953$ disk.dm-0.out‚260533.23876953125 disk.sda3.out‚ 659.0703125 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0( disk./.usagePercent‚78.8498996696233 disk.drbd1260.out‚1.546875 disk.sda.out‚ 659.0703125 disk.sda2.out‚0$ disk.nvme0c0n1.in‚33876.79296875' disk.nvme0n1.out‚260533.23876953125 cpu‚25.302979411910314mY$º4“)/ _wandbruntime‚11006  _runtime‚11006ûØ;bÈ”) $ _timestamp‚1.788498907058396e+09 _runtime‚11006.698869953 _step‚844 & lr-AdamW/pg1‚5.179075270324725e-05 & lr-AdamW/pg2‚5.179075270324725e-05 trainer/global_step‚10299ÍxViÍ•)Ç _step‚844 & lr-AdamW/pg1‚5.179075270324725e-05 & lr-AdamW/pg2‚5.179075270324725e-05 trainer/global_step‚10299 $ _timestamp‚1.788498907058396e+09 _runtime‚11006.698869953Ì0hî4–)/ _wandbruntime‚11006  _runtime‚11006Êd~ì(—)P A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚712368548439718 _runtime‚11006.70024271 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚566 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11037163402240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10586726024704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10585953379840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57335994 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚701819934020176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚730833112377344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10548614419542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 408851043 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10586726024704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚568 trainer/global_step‚10299 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103402737 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 228524901 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚720246368349696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103402735 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 180349160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚810820363580416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 180349160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57335428 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚720246386352640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160738731 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10547843771130 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 228501885 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 180349158 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚799783400611840 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10585953379840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 180349158 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 408874061 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 228501885 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚712367760751946 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228524901 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚701819916980816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚730832321729536 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚730832321729536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846x\HØŽ˜)†P N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160738163 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚720246368349696 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚712368548439718 _runtime‚11006.70024271 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚566 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11037163402240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10586726024704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10585953379840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57335994 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚701819934020176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚730833112377344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10548614419542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 408851043 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10586726024704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚568 trainer/global_step‚10299 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103402737 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 228524901 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚720246368349696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103402735 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 180349160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚810820363580416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 180349160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57335428 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚720246386352640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160738731 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10547843771130 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 228501885 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 180349158 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚799783400611840 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3ñ‹Î ~ 125 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10585953379840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 180349158 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 408874061 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 228501885 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚712367760751946 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228524901 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚701819916980816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚730832321729536 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚730832321729536 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160753110 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 408889628 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103412385 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 408912672 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 408912672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10586958125568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57341283 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228546638 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚556 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚720314086291968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194128896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720314068289024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚558 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 % _timestamp‚1.7884989079690082e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 228523596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚799858772894720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10548844911008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚730901817075712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚846 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 180366032 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 180366032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 228523596 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57340727 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 trainer/global_step‚10299 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10586958125568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 trainer/global_step‚10299 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10586958125568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160753110 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 408889628 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103412385 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 408912672 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 408912672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10586958125568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57341283 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228546638 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚556 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚720314086291968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194128896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720314068289024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚558 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 % _timestamp‚1.7884989079690082e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 228523596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚799858772894720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10548844911008΂¹®4œ)/ _wandbruntime‚11007  _runtime‚11007*(Y{_)Ù 0loss/FRt4/assignment_loss‚1.1146580147392608 epoch‚8 /loss/FRt3/assignment_loss‚3.181281193442967 /loss/FRt3/detection_loss‚3.1161489315007045 /loss/FBt4/detection_loss‚0.4852709933056509 2loss/SRqqt1/assignment_loss‚1.5389506927957493 2loss/SRqqt3/assignment_loss‚1.3762084109565387 1loss/FBt1/assignment_loss‚0.19178327131277417 /loss/FRt4/detection_loss‚1.2648810941506834 /loss/SRqqt1/detection_loss‚2.61697185252394 1loss/SRqqt3/detection_loss‚1.6648174769316728 0loss/FRt2/assignment_loss‚3.1688448609532536 1loss/FBt2/assignment_loss‚0.29595149255042735 1loss/FBt3/assignment_loss‚0.07545287627141128 /loss/FRt1/detection_loss‚2.2828854894531623 0loss/SRqqt2/detection_loss‚2.297266240749094 1loss/SRqqt4/detection_loss‚0.7737020856887102 _runtime‚11007.611804563 .loss/FRt2/detection_loss‚2.287678673053371 /loss/FBt2/detection_loss‚3.0280563831329346 % _timestamp‚1.7884989079716158e+09 .loss/FBt3/detection_loss‚2.471043693522612 trainer/global_step‚10299 %loss/total_loss‚40.33975198846651 2loss/SRqqt4/assignment_loss‚0.8809216621983796 1loss/FBt4/assignment_loss‚0.07974558233382735 _step‚847 0loss/FRt1/assignment_loss‚2.5119843795185997 2loss/SRqqt2/assignment_loss‚1.4039800990786817 /loss/FBt1/detection_loss‚2.2312665383021035ÃHÇdž)Þ trainer/global_step‚10299 %loss/total_loss‚40.33975198846651 2loss/SRqqt4/assignment_loss‚0.8809216621983796 1loss/FBt4/assignment_loss‚0.07974558233382735 _step‚847 0loss/FRt1/assignment_loss‚2.5119843795185997 2loss/SRqqt2/assignment_loss‚1.4039800990786817 /loss/FBt1/detection_loss‚2.2312665383021035 0loss/FRt4/assignment_loss‚1.1146580147392608 epoch‚8 /loss/FRt3/assignment_loss‚3.181281193442967 /loss/FRt3/detection_loss‚3.1161489315007045 /loss/FBt4/detection_loss‚0.4852709933056509 2loss/SRqqt1/assignment_loss‚1.5389506927957493 2loss/SRqqt3/assignment_loss‚1.3762084109565387 1loss/FBt1/assignment_loss‚0.19178327131277417 /loss/FRt4/detection_loss‚1.2648810941506834 /loss/SRqqt1/detection_loss‚2.61697185252394 1loss/SRqqt3/detection_loss‚1.6648174769316728 0loss/FRt2/assignment_loss‚3.1688448609532536 1loss/FBt2/assignment_loss‚0.29595149255042735 1loss/FBt3/assignment_loss‚0.07545287627141128 /loss/FRt1/detection_loss‚2.2828854894531623 0loss/SRqqt2/detection_loss‚2.297266240749094 1loss/SRqqt4/detection_loss‚0.7737020856887102 _runtime‚11007.611804563 .loss/FRt2/detection_loss‚2.287678673053371 /loss/FBt2/detection_loss‚3.0280563831329346 % _timestamp‚1.7884989079716158e+09 .loss/FBt3/detection_loss‚2.471043693522612Ï‚+™8‰Ÿ)‚(:þ Þ§éÔ °å²) proc.memory.availableMB‚ 1118.40234375* proc.memory.percent‚18.003249168395996 network.sent‚ 2.6274485e+07 disk.sda3.out‚ 659.69921875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚260894.06689453125) disk./.usagePercent‚78.86517077504827$ disk./.usageGB‚1175.148036956787 disk.sda3.in‚0 disk.sda.out‚ 659.69921875 disk.sda1.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚33876.79296875 disk.dm-0.in‚33876.79296875 disk.dm-1.out‚ 659.69921875 disk.drbd1260.out‚1.546875" disk.nvme0n1.in‚33876.79296875# proc.memory.rssMB‚ 2949.65234375 cpu‚25.302710426980767 proc.cpu.threads‚17 network.recv‚ 3.639337e+06 disk.drbd1260.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚260894.06689453125$ disk.dm-0.out‚260894.06689453125$ memory_percent‚93.17381381988525î?Ïä’ )‚(:‡ Þ§éÔöžÐ² gpu.0.gpu‚24.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.509' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚3 ˆ¡)‚(:’ í§éÔ”•«­ gpu.0.gpu‚34.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.226' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.245200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þf …¢)‚(:ú í§éÔ»Û÷¯% disk./.usageGB‚1175.1704864501953 disk.sda2.out‚0' disk.nvme0n1.out‚261294.70458984375$ disk.dm-0.out‚261294.70458984375* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚261294.70458984375# proc.memory.rssMB‚ 2949.65234375 cpu‚25.3024278194666 network.sent‚ 2.638678e+07) disk./.usagePercent‚78.86667737938727 disk.sda3.out‚ 660.1953125 disk.dm-1.out‚ 660.1953125 disk.drbd1260.in‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚34005.09765625" disk.nvme0n1.in‚34005.09765625 network.recv‚ 3.653097e+06 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.55078125 disk.sda.out‚ 660.1953125 disk.sda1.out‚0 disk.dm-0.in‚34005.09765625$ memory_percent‚93.17841529846191( proc.memory.availableMB‚ 1117.6484375Û¼û3’£)‚(:‡ ü§éÔ„¨¨® gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.256' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ß«·…¤)‚(:ú ü§éÔ­¥¼¯" disk.nvme0n1.in‚34005.09765625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 661.046875 disk.sda1.out‚0) disk.nvme0c0n1.out‚261887.47412109375 cpu‚25.302179342520816 proc.cpu.threads‚17 network.sent‚ 2.639371e+07) disk./.usagePercent‚78.86510754118731% disk./.usageGB‚1175.1470947265625 disk.dm-1.out‚ 661.046875 disk.sda1.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚34005.09765625) proc.memory.availableMB‚ 1118.14453125 network.recv‚ 3.658582e+06' disk.nvme0n1.out‚261887.47412109375 disk.dm-0.in‚34005.09765625$ disk.dm-0.out‚261887.47412109375 disk.sda.in‚0$ memory_percent‚93.17538738250732# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 disk.sda3.out‚ 661.046875 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.55078125 disk.sda2.in‚0Öñ뤜¥)‚(:‘ ‹¨éÔìÒŒ® gpu.0.gpu‚33.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.694' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚33.93879999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Úý,†¦)‚(:û ‹¨éÔБù²$ disk.nvme0c0n1.in‚34005.09765625) disk.nvme0c0n1.out‚262143.77783203125 disk.dm-0.in‚34005.09765625 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚18.003249168395996" disk.nvme0n1.in‚34005.09765625' disk.nvme0n1.out‚262143.77783203125 disk.sda3.out‚ 661.39453125 disk.sda.in‚0 disk.sda.out‚ 661.39453125 disk.sda1.in‚0# memory_percent‚93.1739091873169 cpu‚25.30191317406836 network.sent‚ 2.6399087e+07$ disk.dm-0.out‚262143.77783203125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1118.38671875 proc.cpu.threads‚17( disk./.usagePercent‚78.8666722592366 disk.dm-1.out‚ 661.39453125 disk.drbd1260.out‚ 1.5546875# proc.memory.rssMB‚ 2949.65234375 network.recv‚ 3.662332e+06# disk./.usageGB‚1175.17041015625œ»¨4§)/ _wandbruntime‚11058  _runtime‚11058qVzÍɨ)à & lr-AdamW/pg1‚4.899973222335492e-05 & lr-AdamW/pg2‚4.899973222335492e-05 trainer/global_step‚10349 % _timestamp‚1.7884989591500955e+09 _runtime‚11058.790512815 _step‚848Æf‘Ω)È & lr-AdamW/pg2‚4.899973222335492e-05 trainer/global_step‚10349 % _timestamp‚1.7884989591500955e+09 _runtime‚11058.790512815 _step‚848 & lr-AdamW/pg1‚4.899973222335492e-05Ð Í«4ª)/ _wandbruntime‚11058  _runtime‚11058øT ò-«)‚P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10636205080064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229590787 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103885135 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161489904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚734267570396160 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚723631365316096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚814640590245376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceÛŒXŒÛ!StatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚734268361043968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚723631383319040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _step‚849 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚723631383319040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 410806663 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚715721940812826 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚715721153125054 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103885137 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 % _timestamp‚1.7884989591512482e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚723631365316096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10598685004650 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 181192858 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57604767 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 410783645 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚554 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚803550817751040 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10597914356238 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10349 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚734267570396160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚814640367968768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11089748554240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚705123238768816 _runtime‚11058.791677118 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚803550841691136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 229613803 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10636977724928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10636205080064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚705123255808176 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161489348 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11089550217728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 181192858 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚734268361043968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 181192860 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57604213 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10636977724928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0³î6´ (¬)‡P N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161489348 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11089550217728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 181192858 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚734268361043968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 181192860 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57604213 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10636977724928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10636205080064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229590787 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103885135 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 161489904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚734267570396160 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚723631365316096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚814640590245376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚734268361043968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚723631383319040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _step‚849 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚723631383319040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 410806663 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚715721940812826 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚715721153125054 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103885137 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 % _timestamp‚1.7884989591512482e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚723631365316096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10598685004650 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 181192858 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57604767 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 410783645 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚554 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚803550817751040 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10597914356238 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10349 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚734267570396160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚814640367968768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11089748554240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚705123238768816 _runtime‚11058.791677118 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚803550841691136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 229613803 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10636977724928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10636205080064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚705123255808176 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516Ñ.j;X4­)/ _wandbruntime‚11059  _runtime‚11059ƒ¬ÔÔ'®)ŒN D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 410822230 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚723699083258368 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229612498 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11090595449344 _step‚850 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10598915496116 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10637982483968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 181209732 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229635540 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚715788220700692 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚705189322243936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚723699065255424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚734336275081216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚723699065255424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚11059.647081802 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10637209825792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10599686144736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 410845274 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 181209732 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 410845274 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚734337065742336 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚803626190033920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161504268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229612498 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11090795869696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚715789008388672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ã ³’¯¯)‘N YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10598915496116 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10637982483968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 181209732 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229635540 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚715788220700692 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚705189322243936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚723699065255424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚734336275081216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚723699065255424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚11059.647081802 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10637209825792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10599686144736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 410845274 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 181209732 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 D>DeviceS®htatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 410845274 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚734337065742336 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚803626190033920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161504268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229612498 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11090795869696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚715789008388672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 410822230 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚723699083258368 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229612498 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11090595449344 _step‚850 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312ÒUÙªþ4°)/ _wandbruntime‚11059  _runtime‚11059¯}(Z±)Ô trainer/global_step‚10349 /loss/FRt4/assignment_loss‚1.206553126082701 /loss/FBt1/detection_loss‚2.1697041907093744 /loss/FBt2/detection_loss‚2.2436902076005936 /loss/FRt1/detection_loss‚2.3867398985710584 .loss/FRt3/detection_loss‚2.712501501495188 _step‚851 0loss/FBt3/assignment_loss‚0.2389314148133046 /loss/FBt4/detection_loss‚0.5940061223809607 0loss/FRt1/assignment_loss‚2.6919122071642625 0loss/FRt2/assignment_loss‚3.0926968004252458 1loss/SRqqt1/assignment_loss‚1.369044241354321 0loss/FBt4/assignment_loss‚0.1250501506381223 /loss/FRt4/detection_loss‚1.5952781001653742 0loss/SRqqt3/detection_loss‚1.915634522990634 2loss/SRqqt3/assignment_loss‚1.3517786197868797 2loss/SRqqt4/assignment_loss‚0.7786114019463244 .loss/FRt2/detection_loss‚2.502183636998801 epoch‚8 0loss/FRt3/assignment_loss‚2.7782501849261196 0loss/FBt1/assignment_loss‚0.5871701965701122 1loss/SRqqt1/detection_loss‚2.2266206199472602 .loss/FBt3/detection_loss‚2.350542038679123 1loss/SRqqt2/assignment_loss‚1.114750742474023 1loss/FBt2/assignment_loss‚0.31972011501784436 % _timestamp‚1.7884989600084286e+09 _runtime‚11059.648594292 0loss/SRqqt2/detection_loss‚2.103673619382522 1loss/SRqqt4/detection_loss‚0.7035953656000935 %loss/total_loss‚39.158639025720258¿[õ_²)Ù .loss/FRt3/detection_loss‚2.712501501495188 _step‚851 0loss/FBt3/assignment_loss‚0.2389314148133046 /loss/FBt4/detection_loss‚0.5940061223809607 0loss/FRt1/assignment_loss‚2.6919122071642625 0loss/FRt2/assignment_loss‚3.0926968004252458 1loss/SRqqt1/assignment_loss‚1.369044241354321 0loss/FBt4/assignment_loss‚0.1250501506381223 /loss/FRt4/detection_loss‚1.5952781001653742 0loss/SRqqt3/detection_loss‚1.915634522990634 2loss/SRqqt3/assignment_loss‚1.3517786197868797 2loss/SRqqt4/assignment_loss‚0.7786114019463244 .loss/FRt2/detection_loss‚2.502183636998801 epoch‚8 0loss/FRt3/assignment_loss‚2.7782501849261196 0loss/FBt1/assignment_loss‚0.5871701965701122 1loss/SRqqt1/detection_loss‚2.2266206199472602 .loss/FBt3/detection_loss‚2.350542038679123 1loss/SRqqt2/assignment_loss‚1.114750742474023 1loss/FBt2/assignment_loss‚0.31972011501784436 % _timestamp‚1.7884989600084286e+09 _runtime‚11059.648594292 0loss/SRqqt2/detection_loss‚2.103673619382522 1loss/SRqqt4/detection_loss‚0.7035953656000935 %loss/total_loss‚39.15863902572025 trainer/global_step‚10349 /loss/FRt4/assignment_loss‚1.206553126082701 /loss/FBt1/detection_loss‚2.1697041907093744 /loss/FBt2/detection_loss‚2.2436902076005936 /loss/FRt1/detection_loss‚2.3867398985710584Ó·RIK’³)‚(:‡ š¨éÔÔÇþ¯ gpu.0.gpu‚44.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚128.653' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.7306 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºòƱ…´)‚(:ú š¨éÔœ€Ð² disk.sda1.in‚0 disk.sda1.out‚0$ proc.memory.availableMB‚1118.375# proc.memory.rssMB‚ 2949.65234375 cpu‚25.301642839632265 network.recv‚ 3.665629e+06$ disk.nvme0c0n1.in‚34005.09765625 disk.sda.out‚ 661.80859375 disk.sda2.in‚0$ memory_percent‚93.17398071289062* proc.memory.percent‚18.003249168395996 network.sent‚ 2.6403939e+07' disk.nvme0n1.out‚262495.15283203125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.90039536369946$ disk./.usageGB‚1175.672908782959" disk.nvme0n1.in‚34005.09765625 disk.dm-0.in‚34005.09765625$ disk.dm-0.out‚262495.15283203125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.5546875 disk.sda2.out‚0) disk.nvme0c0n1.out‚262495.15283203125 disk.sda3.out‚ 661.80859375 disk.dm-1.out‚ 661.80859375­o±’µ)‚(:‡ ©¨éÔÙ†¿® gpu.0.gpu‚26.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.736' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.5472 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùÕe@…¶)‚(:ú ©¨éÔ°Ô§² disk.drbd1260.out‚ 1.55859375 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 3.672493e+06! disk.nvme0n1.in‚ 34129.5078125 disk.sda3.out‚ 662.17578125 disk.dm-1.in‚0 disk.sda2.out‚0$ memory_percent‚93.17402839660645( proc.memory.availableMB‚ 1118.3671875% disk./.usageGB‚1175.6708717346191# disk.nvme0c0n1.in‚ 34129.5078125( disk.nvme0c0n1.out‚262736.0322265625 disk.dm-0.in‚ 34129.5078125# disk.dm-0.out‚262736.0322265625 disk.sda.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2949.65234375 network.sent‚ 2.6514814e+07) disk./.usagePercent‚78.90025865567617 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 662.17578125 disk.sda2.in‚0* proc.memory.percent‚18.003249168395996 cpu‚25.301395960033766& disk.nvme0n1.out‚262736.0322265625 disk.dm-1.out‚ 662.17578125|³ò’·)‚(:‡ ¸¨éÔÛÌŒ¯ gpu.0.gpu‚43.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.107' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ky²6€¸)‚(:õ ¸¨éÔÀ™³& proc.memory.availableMB‚ 1117.59375# proc.memory.rssMB‚ 2949.65234375 proc.cpu.threads‚17 network.sent‚ 2.6528587e+07 network.recv‚ 3.696272e+06) disk./.usagePercent‚78.87492568613295# disk.nvme0c0n1.in‚ 34129.5078125$ disk.dm-0.out‚263136.73974609375 disk.sda3.in‚0 disk.dm-1.out‚ 662.828125 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 34129.5078125 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.55859375 disk.sda.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.003249168395996 cpu‚25.301121948749426% disk./.usageGB‚1175.2933921813965' disk.nvme0n1.out‚263136.73974609375 disk.dm-0.in‚ 34129.5078125 disk.sda.out‚ 662.828125$ memory_percent‚93.17874908447266) disk.nvme0c0n1.out‚263136.73974609375 disk.sda3.out‚ 662.828125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0}*™4¹)/ _wandbruntime‚11109  _runtime‚11109‚¡™Ëº)Å ' lr-AdamW/pg2‚4.6281051306669226e-05 trainer/global_step‚10399 % _timestamp‚1.7884990099905708e+09 _runtime‚11109.631067489 _step‚852 ' lr-AdamW/pg1‚4.6281051306669226e-05)¿XºÐ»)Ê _step‚852 ' lr-AdamW/pg1‚4.6281051306669226e-05 ' lr-AdamW/pg2‚4.6281051306669226e-05 trainer/global_step‚10399 % _timestamp‚1.7884990099905708e+09 _runtime‚11109.631067489Ô ¾+4¼)/ _wandbruntime‚11109  _runtime‚11109ßéç(½)P P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10687229425152 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10686456780288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚737703609710592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 182036560 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104367535 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚818460551119872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚853 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚737702819062784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚719075333185934 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10687229425152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚11109.63273407 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 230702705 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 412716247 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182036560 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚737702819062784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209693696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚737703609710592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚727016380285440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57872287 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚727016362282496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚708426577596176 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚719074545498162 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162239286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚807318234890240 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11142106535936 trainer/global_step‚10399 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230679689 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚727016362282496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230679689 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57871751 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 182036558 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10647984941346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10648755589758 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10647984941346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10648755589758 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚737702819062784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚719075333185934 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10687229425152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚11109.63273407 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 230702705 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 412716247 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182036560 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚737702819062784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209693696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚737703609710592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚727016380285440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57872287 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚727016362282496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚708426577596176 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚719074545498162 TGDeviceStatsMonitor.on_train_batch_start/act,ô;—ive_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162239286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚807318234890240 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11142106535936 trainer/global_step‚10399 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230679689 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚727016362282496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230679689 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57871751 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 182036558ÕnKc4¿)/ _wandbruntime‚11110  _runtime‚11110Z'À)ŒN ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10649756729844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10648986081224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 182053432 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10688234184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚737772314408960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚727084062221824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚727084062221824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚818536756445184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚854 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104377185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚807393583233024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10687461526016 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182053434 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10687461526016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚708492626992576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 % _timestamp‚1.7884990109315848e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 412777876 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 230724442 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185740288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚727084080224768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162254902 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11143358952448 trainer/global_step‚10399 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚737771523747840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11143173212160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10688234184192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚719141613073800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚727084080224768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 230724442 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚708492644031936 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 412777876 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 230701400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182053434 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚11110.572586725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚807393607173120 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57877161 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162254344 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104377183 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 412754832üµU†'Á)‘N S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚737771523747840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11143173212160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10688234184192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚719141613073800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚727084080224768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 230724442 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚708492644031936 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 412777876 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 230701400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182053434 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _runtime‚11110.572586725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚807393607173120 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57877161 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162254344 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104377183 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 412754832 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10649756729844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10648986081224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 182053432 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10688234184192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚737772314408960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚727084062221824 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚727084062221824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚818536756445184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚854 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104377185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚807393583233024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10687461526016 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182053434 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10687461526016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚708492626992576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 % _timestamp‚1.7884990109315848e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 412777876 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 230724442 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185740288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚727084080224768 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162254902 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11143358952448 trainer/global_step‚10399ÖxÙâ¹4Â)/ _wandbruntime‚11110  _runtime‚11110µUôz_Ã)Ù 0loss/FBt4/assignment_loss‚0.0723870742368456 /loss/FRt2/detection_loss‚2.2900758646124317 2loss/SRqqt1/assignment_loss‚0.9243953943090595 1loss/FBt3/assignment_loss‚0.13883403411620518 1loss/SRqqt1/detection_loss‚2.7183673226315044 1loss/SRqqt4/detection_loss‚0.7336579812494548 /loss/FBt4/detection_loss‚0.5076397372968495 /loss/FRt2/assignment_loss‚2.711010788426255 2loss/SRqqt4/assignment_loss‚0.8210159141956817 _runtime‚11110.574956991 1loss/SRqqt3/detection_loss‚2.0004505429416897 %loss/total_loss‚38.87197902262665 epoch‚8 % _timestamp‚1.7884990109342551e+09 _step‚855 trainer/global_step‚10399 .loss/FRt3/assignment_loss‚2.75283051840961 1loss/SRqqt2/assignment_loss‚1.405745888799429 2loss/SRqqt3/assignment_loss‚1.2906200475990772 0loss/FBt1/assignment_loss‚0.7034714108126023 /loss/FRt1/detection_loss‚2.2179994566652637 /loss/FBt2/detection_loss‚2.5711638629436493 /loss/FBt3/detection_loss‚1.8182064782489429 0loss/FRt4/assignment_loss‚1.4075286762189056 /loss/FRt4/detection_loss‚1.3568863403999198 /loss/FRt3/detection_loss‚2.5872870786115527 1loss/SRqqt2/detection_loss‚2.3272991406917574 /loss/FBt1/detection_loss‚2.2099909120135837 0loss/FRt1/assignment_loss‚3.1677946597337723 1loss/FBt2/assignment_loss‚0.13731989746260195¨ëŽdÄ)Þ 0loss/FRt4/assignment_loss‚1.4075286762189056 /loss/FRt4/detection_loss‚1.3568863403999198 /loss/FRt3/detection_loss‚2.5872870786115527 1loss/SRqqt2/detection_loss‚2.3272991406917574 /loss/FBt1/detection_loss‚2.2099909120135837 0loss/FRt1/assignment_loss‚3.1677946597337723 1loss/FBt2/assignment_loss‚0.13731989746260195 0loss/FBt4/assignment_loss‚0.0723870742368456 /loss/FRt2/detection_loss‚2.2900758646124317 2loss/SRqqt1/assignment_loss‚0.9243953943090595 1loss/FBt3/assignment_loss‚0.13883403411620518 1loss/SRqqt1/detection_loss‚2.7183673226315044 1loss/SRqqt4/detection_loss‚0.7336579812494548 /loss/FBt4/detection_loss‚0.5076397372968495 /loss/FRt2/assignment_loss‚2.711010788426255 2loss/SRqqt4/assignment_loss‚0.8210159141956817 _runtime‚11110.574956991 1loss/SRqqt3/detection_loss‚2.0004505429416897 %loss/total_loss‚38.87197902262665 epoch‚8 % _timestamp‚1.7884990109342551e+09 _step‚855 trainer/global_step‚10399 .loss/FRt3/assignment_loss‚2.75283051840961 1loss/SRqqt2/assignment_loss‚1.405745888799429 2loss/SRqqt3/assignment_loss‚1.2906200475990772 0loss/FBt1/assignment_loss‚0.7034714108126023 /loss/FRt1/detection_loss‚2.2179994566652637 /loss/FBt2/detection_loss‚2.5711638629436493 /loss/FBt3/detection_loss‚1.8182064782489429×ïdd³’Å)‚(:‡ ǨéԊ癲 gpu.0.gpu‚48.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚125.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©UÌŽzÆ)‚(:ï ǨéÔة׺ disk.sda2.out‚0 disk.sda3.out‚ 663.078125 disk.dm-1.out‚ 663.078125 disk.sda.in‚0 disk.sda.out‚ 663.078125$ memory_percent‚93.17605495452881 network.sent‚ 2.653391e+07% disk./.usageGB‚1175.2933616638184 disk.dm-0.in‚ 34129.5078125 disk.drbd1260.out‚1.5625* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17 disk.sda1.out‚0( disk.nvme0c0n1.out‚263537.5244140625! disk.nvme0n1.in‚ 34129.5078125# disk.dm-0.out‚263537.5244140625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 cpu‚25.300862054720366 network.recv‚ 3.70009e+06 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 34129.5078125& disk.nvme0n1.out‚263537.5244140625) proc.memory.availableMB‚ 1118.03515625# proc.memory.rssMB‚ 2949.65234375) disk./.usagePercent‚78.87492363807269 disk.sda1.in‚0ËÏ·Ç)‚(:’ Ö¨éÔÈÚ¶® gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.562599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ\UÈ)‚(:ö Ö¨éÔŠˆ° network.sent‚ 2.6641744e+07) disk.nvme0c0n1.out‚263962.17236328125! disk.nvme0n1.in‚ 34129.5078125$ disk.dm-0.out‚263962.17236328125 disk.dm-1.in‚0 disk.dm-1.out‚ 663.8203125 disk.drbd1260.in‚0( proc.memory.availableMB‚ 1118.2890625% disk./.usageGB‚1175.2699241638184' disk.nvme0n1.out‚263962.17236328125 disk.dm-0.in‚ 34129.5078125# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 cpu‚25.300642330795654 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 663.8203125 disk.sda.in‚0 disk.sda.out‚ 663.8203125 disk.sda2.in‚0$ memory_percent‚93.17450523376465 network.recv‚ 3.705938e+06) disk./.usagePercent‚78.87335072778231# disk.nvme0c0n1.in‚ 34129.5078125 disk.drbd1260.out‚1.5625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0Ïâ†jÉ)‚(:„ å¨éÔçÜ©² gpu.0.gpu‚34.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚99.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.862 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M }†Ê)‚(:û å¨éÔ¯‹´ network.recv‚ 3.710695e+06' disk.nvme0n1.out‚264289.79248046875 disk.sda3.in‚0 disk.drbd1260.out‚ 1.56640625 disk.sda.in‚0 disk.sda2.in‚0* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17 network.sent‚ 2.6648005e+07% disk./.usageGB‚1175.2464790344238) disk.nvme0c0n1.out‚264289.79248046875 disk.nvme0n1.in‚ 34241.828125 disk.dm-1.out‚ 664.30078125 disk.drbd1260.in‚0 disk.sda.out‚ 664.30078125 disk.sda1.in‚0) disk./.usagePercent‚78.87177730547688" disk.nvme0c0n1.in‚ 34241.828125 disk.dm-0.in‚ 34241.828125$ disk.dm-0.out‚264289.79248046875 disk.sda1.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1117.77734375# proc.memory.rssMB‚ 2949.65234375 disk.sda3.out‚ 664.30078125 disk.dm-1.in‚0$ memory_percent‚93.17762851715088 cpu‚25.300400452682656~ͧo4Ë)/ _wandbruntime‚11159  _runtime‚11159œf‰<ÉÌ)à % _timestamp‚1.7884990603153136e+09 _runtime‚11159.956087015 _step‚856 & lr-AdamW/pg1‚4.363527652221846e-05 & lr-AdamW/pg2‚4.363527652221846e-05 trainer/global_step‚10449È;tgÎÍ)È _step‚856 & lr-AdamW/pg1‚4.363527652221846e-05 & lr-AdamW/pg2‚4.363527652221846e-05 trainer/global_step‚10449 % _timestamp‚1.7884990603153136e+09 _runtime‚11159.956087015Ø#ü€;4Î)/ _wandbruntime‚11159  _runtime‚11159©ìÄÕÏ)‚P UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104849935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚730401359248896 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10698055526454 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 trainer/global_step‚10449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58140294 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 231791607 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚566 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytesXKÊò3.all.freed‚722427937871270 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10698826174866 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚741138067729408 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162990229 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 414648849 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10736708480512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58140860 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚811085652029440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚741138858377216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚822280269535232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚730401359248896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚722428725559042 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚730401377251840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚741138858377216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 _runtime‚11159.957926391 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182880258 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11194617505792 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 182880258 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚811085675969536 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182880260 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58140294 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 231791607 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚566 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚722427937871270 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10698826174866 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚741138067729408 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162990229 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 414648849 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10736708480512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58140860 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚811085652029440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚741138858377216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚822280269535232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚730401359248896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚722428725559042 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚730401377251840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚741138858377216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 _runtime‚11159.957926391 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182880258 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11194617505792 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 182880258 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚811085675969536 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182880260 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚722495793134888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163005770 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚822356893495296 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182897134 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚822356683814912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚730469059188224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58145649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11195869182976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚722495005446908 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10699827314952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 414710478 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚741206772414464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182897134 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚730469077191168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚730469077191168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182897132 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231790302 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚811161024312320 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚11161.161840144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58146185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10738485884416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚858 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163005232 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104859583 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 231790302 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10699056666332 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 $ _timestamp‚1.788499061520563e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10737713226240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 182897132 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11195683442688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚741207563075584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 231813344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231813344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10449 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10738485884416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚711795948780576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516˜>-8Õ)N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚722495793134888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163005770 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚822356893495296 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182897134 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚822356683814912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚730469059188224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58145649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11195869182976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚722495005446908 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10699827314952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 414710478 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚741206772414464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182897134 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚730469077191168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚730469077191168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182897132 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231790302 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚811161024312320 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚11161.161840144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58146185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10738485884416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚858 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163005232 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104859583 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 231790302 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10699056666332 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 $ _timestamp‚1.788499061520563e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10737713226240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 182897132 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11195683442688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚741207563075584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 231813344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231813344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10449 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10738485884416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚711795948780576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516ÚÂJð4Ö)/ _wandbruntime‚11161  _runtime‚11161Ÿ$™]×)× .loss/FBt2/detection_loss‚2.751995725291116 epoch‚8 0loss/FRt1/assignment_loss‚2.9981556171324195 1loss/SRqqt1/assignment_loss‚1.923929885353731 0loss/SRqqt2/detection_loss‚2.259987407260471 .loss/FBt3/detection_loss‚2.333345837891102 0loss/FBt4/detection_loss‚0.47667973193846924 _step‚859 /loss/FRt3/detection_loss‚2.5405106377359985 $loss/total_loss‚40.6658065434798 2loss/SRqqt4/assignment_loss‚1.0209524106215109 1loss/FBt1/assignment_loss‚0.14200568043770467 trainer/global_step‚10449 % _timestamp‚1.7884990615235074e+09 1loss/FBt3/assignment_loss‚0.10589597775629045 1loss/SRqqt3/detection_loss‚1.8127937203851239 _runtime‚11161.163790739 .loss/FRt2/assignment_loss‚2.97411761539323 .loss/FRt2/detection_loss‚2.510930449840219 /loss/FRt4/detection_loss‚1.4278261902657423 1loss/SRqqt1/detection_loss‚2.7645070008609607 0loss/FRt3/assignment_loss‚2.6786229078834123 2loss/SRqqt2/assignment_loss‚1.5764513090252876 1loss/FBt2/assignment_loss‚0.11347634850595309 1loss/SRqqt4/detection_loss‚0.7909291275997752 /loss/FBt1/detection_loss‚2.3528857032457986 /loss/FRt4/assignment_loss‚1.311162905259566 2loss/SRqqt3/assignment_loss‚1.2046689408156892 1loss/FBt4/assignment_loss‚0.16359998015747593 /loss/FRt1/detection_loss‚2.4303754328227622»–jMbØ)Ü .loss/FRt2/detection_loss‚2.510930449840219 /loss/FRt4/detection_loss‚1.4278261902657423 1loss/SRqqt1/detection_loss‚2.7645070008609607 0loss/FRt3/assignment_loss‚2.6786229078834123 2loss/SRqqt2/assignment_loss‚1.5764513090252876 1loss/FBt2/assignment_loss‚0.11347634850595309 1loss/SRqqt4/detection_loss‚0.7909291275997752 /loss/FBt1/detection_loss‚2.3528857032457986 /loss/FRt4/assignment_loss‚1.311162905259566 2loss/SRqqt3/assignment_loss‚1.2046689408156892 1loss/FBt4/assignment_loss‚0.16359998015747593 /loss/FRt1/detection_loss‚2.4303754328227622 .loss/FBt2/detection_loss‚2.751995725291116 epoch‚8 0loss/FRt1/assignment_loss‚2.9981556171324195 1loss/SRqqt1/assignment_loss‚1.923929885353731 0loss/SRqqt2/detection_loss‚2.259987407260471 .loss/FBt3/detection_loss‚2.333345837891102 0loss/FBt4/detection_loss‚0.47667973193846924 _step‚859 /loss/FRt3/detection_loss‚2.5405106377359985 $loss/total_loss‚40.6658065434798 2loss/SRqqt4/assignment_loss‚1.0209524106215109 1loss/FBt1/assignment_loss‚0.14200568043770467 trainer/global_step‚10449 % _timestamp‚1.7884990615235074e+09 1loss/FBt3/assignment_loss‚0.10589597775629045 1loss/SRqqt3/detection_loss‚1.8127937203851239 _runtime‚11161.163790739 .loss/FRt2/assignment_loss‚2.97411761539323ÛiWX†’Ù)‚(:‡ ƒ©éÔÕ–Ë® gpu.0.gpu‚38.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.792' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ°ÆÑzÚ)‚(:ï ƒ©éÔ‚·šµ disk.sda.out‚ 665.12890625 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 34241.828125 disk.nvme0n1.in‚ 34241.828125 disk.sda3.out‚ 665.12890625 cpu‚25.299883012919963 proc.cpu.threads‚17( disk./.usagePercent‚78.8749259421405% disk./.usageGB‚1175.2933959960938 disk.dm-1.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚264931.181640625 disk.dm-0.in‚ 34241.828125 disk.sda3.in‚0& proc.memory.availableMB‚ 1117.59375 disk.dm-1.out‚ 665.12890625 disk.drbd1260.in‚0 disk.sda1.in‚0$ memory_percent‚93.17874908447266# proc.memory.rssMB‚ 2949.65234375 network.sent‚ 2.6750449e+07 network.recv‚ 3.72141e+06 disk.drbd1260.out‚ 1.5703125 disk.sda1.out‚0% disk.nvme0n1.out‚264931.181640625" disk.dm-0.out‚264931.181640625* proc.memory.percent‚18.003249168395996 disk.sda.in‚0ÆžÜG‘Û)‚(:† ’©éÔí”Á­ gpu.0.gpu‚32.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.465' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.293 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å~r|Ü)‚(:ñ ’©éÔ“…ï disk.sda2.in‚0" disk.nvme0c0n1.in‚ 34241.828125 disk.dm-0.in‚ 34241.828125# proc.memory.rssMB‚ 2949.65234375 disk.sda3.out‚ 665.5546875 disk.dm-1.in‚0' disk.nvme0c0n1.out‚265379.802734375" disk.dm-0.out‚265379.802734375$ memory_percent‚93.17877292633057) proc.memory.availableMB‚ 1117.58984375* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17 network.sent‚ 2.6847734e+07 disk.drbd1260.out‚ 1.5703125 disk.sda1.out‚0% disk.nvme0n1.out‚265379.802734375 cpu‚25.299621255434705 network.recv‚ 3.727034e+06) disk./.usagePercent‚78.87335482390286% disk./.usageGB‚1175.2699851989746 disk.dm-1.out‚ 665.5546875 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 34241.828125 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 665.5546875<óðeÝ)‚(:’ ¡©éÔÄâÁ² gpu.0.gpu‚38.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.596' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.119200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)‚&ë‡Þ)‚(:ü ¡©éÔñ슷 disk.sda1.out‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚34308.26171875& disk.nvme0n1.out‚265831.7177734375 disk.dm-1.in‚0* proc.memory.percent‚18.003249168395996( disk.nvme0c0n1.out‚265831.7177734375# disk.dm-0.out‚265831.7177734375 disk.sda3.out‚ 665.98828125 disk.sda.out‚ 665.98828125# proc.memory.rssMB‚ 2949.65234375 cpu‚25.299388293556884 proc.cpu.threads‚17 network.recv‚ 3.731008e+06" disk.nvme0n1.in‚34308.26171875 disk.dm-1.out‚ 665.98828125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.57421875 disk.sda.in‚0$ memory_percent‚93.17746162414551( proc.memory.availableMB‚ 1117.8046875 network.sent‚ 2.6853226e+07 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚34308.26171875 disk.sda3.in‚0) disk./.usagePercent‚78.87354119738778$ disk./.usageGB‚1175.272762298584gíÐk4ß)/ _wandbruntime‚11210  _runtime‚11210ÁüÕ›Éà)à & lr-AdamW/pg1‚4.106295924550214e-05 & lr-AdamW/pg2‚4.106295924550214e-05 trainer/global_step‚10499 % _timestamp‚1.7884991107220361e+09 _runtime‚11210.362482541 _step‚860ð‘²@Îá)È & lr-AdamW/pg2‚4.106295924550214e-05 trainer/global_step‚10499 % _timestamp‚1.7884991107220361e+09 _runtime‚11210.362482541 _step‚860 & lr-AdamW/pg1‚4.106295924550214e-05Üê]©æ4â)/ _wandbruntime‚11210  _runtime‚11210Ó°Þ(ã)‚P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚563 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚725782117932150 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183723960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚744573316396032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58408667 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚814853069168640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208822272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10787732825600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 416604469 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 163741002 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚826100353376768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 trainer/global_step‚10499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 416581451 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11247284208128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚565 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105332337 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232880509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚733786374218240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183723958 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10748896759974 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58409230 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚826100586139136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚715033204132816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 183723960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚744573316396032 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 163741567 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884991107232344e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚733786374218240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚733786356215296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _runtime‚11210.363705085^IG{'ä)‡P ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 trainer/global_step‚10499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 416581451 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11247284208128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚565 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105332337 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232880509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚733786374218240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183723958 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10748896759974 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58409230 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚826100586139136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚715033204132816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 183723960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚744573316396032 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 163741567 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884991107232344e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚733786374218240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚733786356215296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 _runtime‚11210.363705085 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚563 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚725782117932150 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183723960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚744573316396032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58408667 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚814853069168640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208822272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10787732825600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 416604469 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 N@DeviceStatsMonitor.on_train#c%’_batch_start/inactive_split.all.freed‚ 163741002 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚826100353376768ÝØ>_4å)/ _wandbruntime‚11211  _runtime‚11211ÏÁ¸'æ)ŒN MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚725848397820016 _runtime‚11211.256858657 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183740834 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚733854074157568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183740832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 232902246 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416643080 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163756531 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚733854074157568 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10787964926464 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10749897900060 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183740832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58413985 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚733854056154624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚814928417511424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚814928441451520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 416620036 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚744642811742208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚715099270568576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 183740834 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58414546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 416620036 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11248526790144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚715099287607936 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚744642811742208 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 232902246 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚826176968241664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚744642021081088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10499 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163755968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚715099287607936 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚744642811742208 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 232902246 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚826176968241664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚744642021081088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10499 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163755968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183740834 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚733854074157568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183740832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 232902246 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416643080 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163756531 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚733854074157568 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10787964926464 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10749897900060 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183740832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58413985 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚733854056154624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚814928417511424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚814928441451520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 416620036 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚744642811742208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚715099270568576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046ÞXŒ:¦4è)/ _wandbruntime‚11211  _runtime‚11211Õ<bé)Ü 0loss/FRt1/assignment_loss‚3.2018070792158446 2loss/SRqqt2/assignment_loss‚1.2305877421672144 1loss/FBt4/assignment_loss‚0.12272682263023332 0loss/SRqqt2/detection_loss‚2.307415157556534 /loss/FBt1/detection_loss‚2.5441385755936303 2loss/SRqqt3/assignment_loss‚1.5534938619016332 2loss/FBt3/assignment_loss‚0.051986381870663935 /loss/FRt3/detection_loss‚2.6875916288958654 &loss/total_loss‚38.368992607743564 _step‚863 % _timestamp‚1.7884991116181386e+09 /loss/FRt1/detection_loss‚2.4504189616224417 .loss/FBt2/detection_loss‚2.995944764879015 1loss/FBt2/assignment_loss‚0.10885982790725596 .loss/FRt4/detection_loss‚1.343574237367007 2loss/SRqqt4/assignment_loss‚0.8527964076695397 /loss/FBt3/detection_loss‚1.8711792528629303 /loss/FBt4/detection_loss‚0.6265963926148342 epoch‚8 trainer/global_step‚10499 0loss/FRt3/assignment_loss‚2.3653717283849365 0loss/FRt4/assignment_loss‚0.8110840857868463 1loss/SRqqt1/assignment_loss‚1.426350740655776 1loss/FBt1/assignment_loss‚0.15526810949729905 0loss/SRqqt1/detection_loss‚2.188213869448631 1loss/SRqqt4/detection_loss‚0.7844770865757829 0loss/FRt2/assignment_loss‚2.5346225102742515 1loss/SRqqt3/detection_loss‚1.9889753411741966 _runtime‚11211.258490211 /loss/FRt2/detection_loss‚2.1655120411912017ûç‚gê)á 1loss/FBt2/assignment_loss‚0.10885982790725596 .loss/FRt4/detection_loss‚1.343574237367007 2loss/SRqqt4/assignment_loss‚0.8527964076695397 /loss/FBt3/detection_loss‚1.8711792528629303 /loss/FBt4/detection_loss‚0.6265963926148342 epoch‚8 trainer/global_step‚10499 0loss/FRt3/assignment_loss‚2.3653717283849365 0loss/FRt4/assignment_loss‚0.8110840857868463 1loss/SRqqt1/assignment_loss‚1.426350740655776 1loss/FBt1/assignment_loss‚0.15526810949729905 0loss/SRqqt1/detection_loss‚2.188213869448631 1loss/SRqqt4/detection_loss‚0.7844770865757829 0loss/FRt2/assignment_loss‚2.5346225102742515 1loss/SRqqt3/detection_loss‚1.9889753411741966 _runtime‚11211.258490211 /loss/FRt2/detection_loss‚2.1655120411912017 0loss/FRt1/assignment_loss‚3.2018070792158446 2loss/SRqqt2/assignment_loss‚1.2305877421672144 1loss/FBt4/assignment_loss‚0.12272682263023332 0loss/SRqqt2/detection_loss‚2.307415157556534 /loss/FBt1/detection_loss‚2.5441385755936303 2loss/SRqqt3/assignment_loss‚1.5534938619016332 2loss/FBt3/assignment_loss‚0.051986381870663935 /loss/FRt3/detection_loss‚2.6875916288958654 &loss/total_loss‚38.368992607743564 _step‚863 % _timestamp‚1.7884991116181386e+09 /loss/FRt1/detection_loss‚2.4504189616224417 .loss/FBt2/detection_loss‚2.995944764879015ßåL+žë)‚(:’ °©éÔಽ® gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚152.792' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.558400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ]¡†ì)‚(:û °©éÔõçÖ° network.recv‚ 3.735281e+06 disk.dm-1.out‚ 666.68359375 disk.drbd1260.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚266280.2451171875# disk.dm-0.out‚266280.2451171875* proc.memory.percent‚18.003249168395996 network.sent‚ 2.6859184e+07$ disk./.usageGB‚1175.296230316162 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚34365.08203125# proc.memory.rssMB‚ 2949.65234375) disk./.usagePercent‚78.87511615573844 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 666.68359375" disk.nvme0n1.in‚34365.08203125 disk.dm-0.in‚34365.08203125$ memory_percent‚93.18075180053711 cpu‚25.299130203893338 disk.sda3.out‚ 666.68359375 disk.drbd1260.out‚ 1.57421875( disk.nvme0c0n1.out‚266280.2451171875' proc.memory.availableMB‚ 1117.265625 proc.cpu.threads‚17jÆ’í)‚(:‡ ¿©éÔœíʱ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚125.873' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1746 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û½EÈ‚î)‚(:÷ ¿©éÔ»»é³ disk.sda2.out‚0) disk.nvme0c0n1.out‚266631.76708984375" proc.memory.availableMB‚1117.5# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 cpu‚25.298873957688972 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚34365.08203125 disk.dm-0.in‚34365.08203125 proc.cpu.threads‚17 network.recv‚ 3.741237e+06" disk.nvme0n1.in‚34365.08203125' disk.nvme0n1.out‚266631.76708984375$ disk.dm-0.out‚266631.76708984375# memory_percent‚93.1793212890625 network.sent‚ 2.6967216e+07) disk./.usagePercent‚78.87511769178363% disk./.usageGB‚1175.2962532043457 disk.sda3.in‚0 disk.sda3.out‚ 666.91796875 disk.dm-1.out‚ 666.91796875 disk.sda.in‚0 disk.sda.out‚ 666.91796875 disk.drbd1260.out‚1.578125 disk.sda2.in‚0°¦g’ï)‚(:‡ ΩéÔۡį gpu.0.gpu‚24.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚103.887' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7774 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èYvu‰ð)‚(:þ ΩéÔ“¨ã´ disk.sda1.out‚0) proc.memory.availableMB‚ 1117.74609375% disk./.usageGB‚1175.2962341308594' disk.nvme0n1.out‚266912.12646484375$ disk.dm-0.out‚266912.12646484375 disk.sda3.in‚0 disk.sda3.out‚ 667.27734375 disk.sda.in‚0* proc.memory.percent‚18.003249168395996 network.sent‚ 2.6972099e+07 disk.sda2.in‚0) disk.nvme0c0n1.out‚266912.12646484375 disk.dm-0.in‚34365.08203125 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2949.65234375 proc.cpu.threads‚17 disk.sda2.out‚0" disk.nvme0n1.in‚34365.08203125 disk.dm-1.out‚ 667.27734375 disk.drbd1260.out‚1.578125 disk.sda.out‚ 667.27734375$ memory_percent‚93.17781925201416 cpu‚25.298628131064433 network.recv‚ 3.74462e+06) disk./.usagePercent‚78.87511641174598$ disk.nvme0c0n1.in‚34365.08203125 disk.dm-1.in‚0 disk.sda1.in‚0jU*´4ñ)/ _wandbruntime‚11261  _runtime‚11261Ž;³uËò)Å _step‚864 ' lr-AdamW/pg1‚3.8564635543585314e-05 ' lr-AdamW/pg2‚3.8564635543585314e-05 trainer/global_step‚10549 % _timestamp‚1.7884991621873999e+09 _runtime‚11261.828121658^9QÐó)Ê ' lr-AdamW/pg1‚3.8564635543585314e-05 ' lr-AdamW/pg2‚3.8564635543585314e-05 trainer/global_step‚10549 % _timestamp‚1.7884991621873999e+09 _runtime‚11261.828121658 _step‚864à7d à4ô)/ _wandbruntime‚11261  _runtime‚11261G\@Aõ)€P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 184567660 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58677308 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 164492596 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748008565062656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚729134722617486 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚10549 _step‚865 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 164492043 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11300071161856 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10798967345082 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚829920581409792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚737171353181696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚737171371184640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚737171371184640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 418514053 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58677859 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105814735 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚11261.83022942 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 184567660 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10837211880960 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 233946395 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11299881213952 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 $ _timestamp‚1.788499162189373e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 233969411 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚729135510305258 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489¾üº (ö)…P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 418514053 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58677859 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105814735 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚11261.83022942 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 184567660 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10837211880960 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 233946395 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11299881213952 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 $ _timestamp‚1.788499162189373e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 233969411 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚729135510305258 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 184567660 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58677308 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 164492596 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748008565062656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚729134722617486 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚10549 _step‚865 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 164492043 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11300071161856 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10798967345082 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚829920581409792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚737171353181696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚737171371184640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189947904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚737171371184640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ᚥ(‡4÷)/ _wandbruntime‚11262  _runtime‚11262÷m/'ø)ŒN C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11301132825600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚718402592356576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚818695834650624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10799197836548 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 418552638 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 184584534 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 233991148 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10838216626688 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10799968485168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚11262.728266162 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚829996777541632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164507022 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚737239071123968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚737239071123968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚866 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105824383 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 164507560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748078060408832 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10838989284864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 233991148 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚818695858590720 % _timestamp‚1.7884991630871456e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚748077269747712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418575682 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58683175 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11300942891008 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚737239053121024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚829996991416320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 418575682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚748078060408832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10838216626688 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 233968106 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚866 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105824383 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 164507560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748078060408832 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10838989284864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 233991148 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚818695858590720 % _timestamp‚1.7884991630871456e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚748077269747712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418575682 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58683175 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11300942891008 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚737239053121024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚829996991416320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 418575682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚748078060408832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10838216626688 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 233968106 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 233991148 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10838216626688 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10799968485168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚11262.728266162 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚829996777541632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164507022 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚737239071123968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚737239071123968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 F®4…*/ _wandbruntime‚11313  _runtime‚11313#qäÂȆ* trainer/global_step‚10599 $ _timestamp‚1.788499213702914e+09 _runtime‚11313.343274083 _step‚868 & lr-AdamW/pg1‚3.614082606338273e-05 & lr-AdamW/pg2‚3.614082606338273e-05®C:͇*Ç $ _timestamp‚1.788499213702914e+09 _runtime‚11313.343274083 _step‚868 & lr-AdamW/pg1‚3.614082606338273e-05 & lr-AdamW/pg2‚3.614082606338273e-05 trainer/global_step‚10599ä¢D[4ˆ*/ _wandbruntime‚11313  _runtime‚113137¡(‰*‚P [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚740556368151040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58946075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58945528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚732488902678366 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10848267281778 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚721639847708816 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11352638668288 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10888236226048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚751444604377088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10888236226048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚740556350148096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165242663 _step‚869 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10887463581184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 trainer/global_step‚10599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚751444604377088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106297135 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚833740764391936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚751443813729280 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165243212 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235058313 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185411358 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11352837004800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚751443813729280 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106297137 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 235058313 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚822387927387136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚740556350148096 % _timestamp‚1.7884992137039802e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚740556368151040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚833740542115328ßztÏ (Š*‡P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚751443813729280 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165243212 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235058313 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185411358 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11352837004800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚751443813729280 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106297137 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 235058313 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚822387927387136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚740556350148096 % _timestamp‚1.7884992137039802e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚740556368151040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚833740542115328 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚740556368151040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58946075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58945528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚732488902678366 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10848267281778 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚721639847708816 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11352638668288 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10888236226048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚751444604377088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10888236226048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚740556350148096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165242663 _step‚869 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10887463581184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 trainer/global_step‚10599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚751444604377088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106297135 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚833740764391936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0å­Eêo4‹*/ _wandbruntime‚11314  _runtime‚11314¸ßt¯oŒ*‹N L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165257576 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚833816924018176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10850039070276 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10889240985088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10849268421656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/resj|Ô6¢#erved_bytes.large_pool.current‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235057008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185428232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235080050 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106306783 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 420508284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58951343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58950793 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235057008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚833817150475776 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚732555182566232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 420485240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235080050 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185428232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11353874745856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 trainer/global_step‚10599 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚740624050087424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10889240985088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚751512518414336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚822463251789824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚740624068090368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106306785 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 185428234 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 4205082844KûT'*N F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 trainer/global_step‚10599 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚740624050087424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10889240985088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚751512518414336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚822463251789824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚740624068090368 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106306785 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 185428234 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 420508284 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165257576 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚833816924018176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10850039070276 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10889240985088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10849268421656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235057008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185428232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235080050 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106306783 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 420508284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58951343 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58950793 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235057008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚833817150475776 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚732555182566232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 420485240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235080050 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185428232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11353874745856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825æðtÍ;4Ž*/ _wandbruntime‚11314  _runtime‚11314Ò&6_*Ù /loss/FRt2/assignment_loss‚3.047997566961473 0loss/FBt4/assignment_loss‚0.1335908174871065 epoch‚8 % _timestamp‚1.7884992145607903e+09 2loss/SRqqt1/assignment_loss‚1.2186250279380664 1loss/FBt3/assignment_loss‚0.15461423900269294 $loss/total_loss‚39.5030845340226 0loss/FRt4/assignment_loss‚1.1874807675679524 1loss/SRqqt2/assignment_loss‚1.263892376864398 .loss/FRt2/detection_loss‚2.510307244970396 /loss/FRt4/detection_loss‚1.4698941105355818 1loss/FBt2/assignment_loss‚0.32037200639816393 /loss/FRt1/detection_loss‚2.5517815897862115 1loss/SRqqt1/detection_loss‚2.1060157128282495 /loss/FBt3/detection_loss‚2.2018590807914733 1loss/SRqqt4/detection_loss‚0.8365508373446611 /loss/FBt1/detection_loss‚2.0453669826189675 /loss/FBt2/detection_loss‚2.0334112693866095 trainer/global_step‚10599 _step‚871 2loss/SRqqt3/assignment_loss‚1.4314162686224217 0loss/FBt1/assignment_loss‚0.5547108721350216 /loss/FRt3/detection_loss‚3.1088642154274315 1loss/SRqqt3/detection_loss‚1.7155628618986711 2loss/SRqqt4/assignment_loss‚0.8230807681054789 1loss/SRqqt2/detection_loss‚2.4787593108636363 /loss/FBt4/detection_loss‚0.5339907025627326 _runtime‚11314.200957781 0loss/FRt3/assignment_loss‚3.1328559312327156 0loss/FRt1/assignment_loss‚2.6420839726924896Ù‘d*Þ 1loss/SRqqt4/detection_loss‚0.8365508373446611 /loss/FBt1/detection_loss‚2.0453669826189675 /loss/FBt2/detection_loss‚2.0334112693866095 trainer/global_step‚10599 _step‚871 2loss/SRqqt3/assignment_loss‚1.4314162686224217 0loss/FBt1/assignment_loss‚0.5547108721350216 /loss/FRt3/detection_loss‚3.1088642154274315 1loss/SRqqt3/detection_loss‚1.7155628618986711 2loss/SRqqt4/assignment_loss‚0.8230807681054789 1loss/SRqqt2/detection_loss‚2.4787593108636363 /loss/FBt4/detection_loss‚0.5339907025627326 _runtime‚11314.200957781 0loss/FRt3/assignment_loss‚3.1328559312327156 0loss/FRt1/assignment_loss‚2.6420839726924896 /loss/FRt2/assignment_loss‚3.047997566961473 0loss/FBt4/assignment_loss‚0.1335908174871065 epoch‚8 % _timestamp‚1.7884992145607903e+09 2loss/SRqqt1/assignment_loss‚1.2186250279380664 1loss/FBt3/assignment_loss‚0.15461423900269294 $loss/total_loss‚39.5030845340226 0loss/FRt4/assignment_loss‚1.1874807675679524 1loss/SRqqt2/assignment_loss‚1.263892376864398 .loss/FRt2/detection_loss‚2.510307244970396 /loss/FRt4/detection_loss‚1.4698941105355818 1loss/FBt2/assignment_loss‚0.32037200639816393 /loss/FRt1/detection_loss‚2.5517815897862115 1loss/SRqqt1/detection_loss‚2.1060157128282495 /loss/FBt3/detection_loss‚2.2018590807914733çÔÝ‚’‘*‚(:‡ ™ªéÔ“á° gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.909' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.5818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aô=@‰’*‚(:þ ™ªéÔš¤Ü± disk.sda3.out‚ 669.41015625) proc.memory.availableMB‚ 1117.76171875 network.sent‚ 2.7103623e+07 disk.sda1.out‚0" disk.nvme0n1.in‚34481.64453125& disk.nvme0n1.out‚268854.2939453125 disk.dm-1.out‚ 669.41015625$ memory_percent‚93.17772388458252# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996) disk./.usagePercent‚78.82861801930943% disk./.usageGB‚1174.6033744812012 disk.sda.in‚0 disk.sda.out‚ 669.41015625( disk.nvme0c0n1.out‚268854.2939453125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 network.recv‚ 3.768025e+06 disk.drbd1260.out‚ 1.58984375 cpu‚25.297314017634434 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚34481.64453125 disk.dm-0.in‚34481.64453125# disk.dm-0.out‚268854.2939453125£]Ù’“*‚(:‡ ¨ªéÔ³†Ù® gpu.0.gpu‚47.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.243' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ=Y‰”*‚(:þ ¨ªéÔ„íÿ± disk.sda.in‚0# proc.memory.rssMB‚ 2949.65234375 network.sent‚ 2.7212167e+07 network.recv‚ 3.774656e+06& disk.nvme0n1.out‚269046.9501953125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 669.96484375 disk.drbd1260.out‚ 1.58984375$ memory_percent‚93.17920207977295) disk./.usagePercent‚78.82862134740739% disk./.usageGB‚1174.6034240722656 disk.sda1.in‚0( disk.nvme0c0n1.out‚269046.9501953125) proc.memory.availableMB‚ 1117.51953125 cpu‚25.297031759607016 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚34612.78515625 disk.dm-0.in‚34612.78515625 disk.sda3.out‚ 669.96484375 disk.sda.out‚ 669.96484375* proc.memory.percent‚18.003249168395996$ disk.nvme0c0n1.in‚34612.78515625# disk.dm-0.out‚269046.9501953125 disk.drbd1260.in‚0 Ÿr’•*‚(:‡ ·ªéÔ€ý ­ gpu.0.gpu‚23.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.819' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5638 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZßÜ¡†–*‚(:û ·ªéÔ­“ã°% disk.nvme0n1.out‚269360.337890625" disk.dm-0.out‚269360.337890625 disk.dm-1.out‚ 670.30078125 disk.drbd1260.in‚0 disk.sda.out‚ 670.30078125% disk./.usageGB‚1172.9524765014648 disk.sda.in‚0) proc.memory.availableMB‚ 1117.27734375* proc.memory.percent‚18.003249168395996 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚34612.78515625' disk.nvme0c0n1.out‚269360.337890625 disk.dm-0.in‚34612.78515625 disk.drbd1260.out‚ 1.58984375$ memory_percent‚93.18068027496338# proc.memory.rssMB‚ 2949.65234375 cpu‚25.296757126758656 network.sent‚ 2.7219255e+07 network.recv‚ 3.778058e+06) disk./.usagePercent‚78.71782487069366 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 670.30078125 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.in‚34612.78515625hÞ”Ô4—*/ _wandbruntime‚11361  _runtime‚11361-¨Ëɘ*à _runtime‚11361.136813709 _step‚872 & lr-AdamW/pg1‚3.379203592315723e-05 & lr-AdamW/pg2‚3.379203592315723e-05 trainer/global_step‚10649 % _timestamp‚1.7884992614965005e+09è ›öΙ*È & lr-AdamW/pg1‚3.379203592315723e-05 & lr-AdamW/pg2‚3.379203592315723e-05 trainer/global_step‚10649 % _timestamp‚1.7884992614965005e+09 _runtime‚11361.136813709 _step‚872èèlñg4š*/ _wandbruntime‚11361  _runtime‚11361#` ”›*‚P N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚743941365117440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚724943186536176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚743941347114496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚837560501769216 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165993036 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚743941347114496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 186255060 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165992493 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10898337866886 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10937715281408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚735841507363702 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11405181182976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 236124199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10938487926272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10938487926272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚826155320586240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚11361.137934631 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 422379257 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10899108515298 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106779537 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59212958 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚754879062395904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚724943169496816 XJDeviceStatsMonitor.on_traina±6žt_batch_start/requested_bytes.small_pool.current‚ 770648412 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚754879853043712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚754879062395904 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚837560719851520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _step‚873 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236124199 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚10649 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚743941365117440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚724943186536176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚743941347114496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚837560501769216 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165993036 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚743941347114496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 186255060 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165992493 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10898337866886 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10937715281408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚735841507363702 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11405181182976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 236124199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10938487926272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10938487926272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚826155320586240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚11361.137934631 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 422379257 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10899108515298 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106779537 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59212958 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚754879062395904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚724943169496816 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚754879853043712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚754879062395904 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚837560719851520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _step‚873 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236124199 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218082304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚10649 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 422440886 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166007405 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 186271932 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 236168952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10938720027136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚826230668929024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10900109655384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 236145910 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚826230692869120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 422440886 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 422417842 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11406227794944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚744009047053824 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚725009252971936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788499262327878e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 186271932 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106789183 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚754948557742080 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10939492685312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚735908574939340 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚744009065056768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236168952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10939492685312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚735909362627320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186271934 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106789185 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚11361.968928901 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10899339006764 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59218772 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚837636896723968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚754948557742080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 236145910 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚837637121084416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚725009235932576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚744009047053824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11406428215296?­Ÿ‡!Ÿ*N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59218772 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚837636896723968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚754948557742080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 236145910 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚837637121084416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚725009235932576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚744009047053824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11406428215296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59218222 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 _step‚874 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚754947767080960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 422440886 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166007405 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 186271932 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 236168952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10938720027136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚826230668929024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10900109655384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 236145910 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚826230692869120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 422440886 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 422417842 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11406227794944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚744009047053824 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚725009252971936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788499262327878e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 186271932 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106789183 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚754948557742080 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10939492685312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚735908574939340 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚744009065056768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236168952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10939492685312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚735909362627320 OmSìÿD;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186271934 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106789185 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚11361.968928901 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10899339006764 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005ê_ £ñ4 */ _wandbruntime‚11361  _runtime‚11361¹¾Î£Y¡*Ó 0loss/FRt2/assignment_loss‚2.9469159462235193 2loss/SRqqt2/assignment_loss‚1.6395501377681891 2loss/SRqqt3/assignment_loss‚1.4707589390809122 0loss/SRqqt4/assignment_loss‚1.02938225477313 0loss/FBt1/assignment_loss‚0.2904541357711423 .loss/FRt3/detection_loss‚2.686080527305603 trainer/global_step‚10649 _step‚875 1loss/SRqqt3/detection_loss‚1.6413032109646692 /loss/FBt1/detection_loss‚2.4981486002604165 /loss/FBt3/detection_loss‚2.1898279104913985 .loss/FRt1/detection_loss‚2.226527431048453 .loss/FBt2/detection_loss‚2.309472934405009 # _timestamp‚1.78849926233037e+09 _runtime‚11361.970506644 1loss/FBt3/assignment_loss‚0.12142407002345733 0loss/FBt4/assignment_loss‚0.1200141061736518 0loss/SRqqt1/detection_loss‚2.710314427984172 0loss/FRt1/assignment_loss‚2.8084358175595603 /loss/FBt4/detection_loss‚0.6051880751877928 %loss/total_loss‚39.88717401224228 0loss/FRt4/assignment_loss‚1.3414866832586436 2loss/SRqqt1/assignment_loss‚1.2393762994686077 /loss/FRt4/detection_loss‚1.4294179608615545 1loss/FBt2/assignment_loss‚0.16124632406035744 0loss/FRt3/assignment_loss‚2.1638733983039855 .loss/FRt2/detection_loss‚3.229916775962667 0loss/SRqqt2/detection_loss‚2.389596751332283 1loss/SRqqt4/detection_loss‚0.6384612939731088 epoch‚8­˜mÛ^¢*Ø 0loss/FRt4/assignment_loss‚1.3414866832586436 2loss/SRqqt1/assignment_loss‚1.2393762994686077 /loss/FRt4/detection_loss‚1.4294179608615545 1loss/FBt2/assignment_loss‚0.16124632406035744 0loss/FRt3/assignment_loss‚2.1638733983039855 .loss/FRt2/detection_loss‚3.229916775962667 0loss/SRqqt2/detection_loss‚2.389596751332283 1loss/SRqqt4/detection_loss‚0.6384612939731088 epoch‚8 0loss/FRt2/assignment_loss‚2.9469159462235193 2loss/SRqqt2/assignment_loss‚1.6395501377681891 2loss/SRqqt3/assignment_loss‚1.4707589390809122 0loss/SRqqt4/assignment_loss‚1.02938225477313 0loss/FBt1/assignment_loss‚0.2904541357711423 .loss/FRt3/detection_loss‚2.686080527305603 trainer/global_step‚10649 _step‚875 1loss/SRqqt3/detection_loss‚1.6413032109646692 /loss/FBt1/detection_loss‚2.4981486002604165 /loss/FBt3/detection_loss‚2.1898279104913985 .loss/FRt1/detection_loss‚2.226527431048453 .loss/FBt2/detection_loss‚2.309472934405009 # _timestamp‚1.78849926233037e+09 _runtime‚11361.970506644 1loss/FBt3/assignment_loss‚0.12142407002345733 0loss/FBt4/assignment_loss‚0.1200141061736518 0loss/SRqqt1/detection_loss‚2.710314427984172 0loss/FRt1/assignment_loss‚2.8084358175595603 /loss/FBt4/detection_loss‚0.6051880751877928 %loss/total_loss‚39.88717401224228ëSú즒£*‚(:‡ ƪéÔ’ѯ gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓêC¼¤*‚(:ô ƪéÔÆ„ú° disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚34612.78515625 disk.drbd1260.out‚1.59375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚269908.880859375' proc.memory.availableMB‚ 1117.421875# proc.memory.rssMB‚ 2949.65234375 network.recv‚ 3.781835e+06% disk./.usageGB‚1173.0030822753906 disk.dm-0.in‚34612.78515625$ disk.nvme0c0n1.in‚34612.78515625# memory_percent‚93.1797981262207 cpu‚25.296454779373946 proc.cpu.threads‚17" disk.dm-0.out‚269908.880859375 disk.sda3.in‚0 disk.sda3.out‚ 670.63671875 disk.dm-1.in‚0 disk.dm-1.out‚ 670.63671875* proc.memory.percent‚18.003249168395996 network.sent‚ 2.7222476e+07( disk./.usagePercent‚78.7212210666429% disk.nvme0n1.out‚269908.880859375 disk.drbd1260.in‚0 disk.sda.out‚ 670.63671875s+Ñ‘¥*‚(:† ÕªéÔ’ÕŒ® gpu.0.gpu‚44.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.685' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.937 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â…N冦*‚(:û ÕªéÔ¾ú° disk.sda3.in‚0 disk.sda3.out‚ 671.03515625 disk.dm-1.in‚0$ memory_percent‚93.18127632141113) disk.nvme0c0n1.out‚270189.31884765625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1117.1796875) disk./.usagePercent‚78.73459208414873 disk.sda1.in‚0 disk.dm-1.out‚ 671.03515625# proc.memory.rssMB‚ 2949.65234375* proc.memory.percent‚18.003249168395996 cpu‚25.29615792321238 proc.cpu.threads‚17 network.recv‚ 3.788876e+06$ disk./.usageGB‚1173.202320098877 disk.drbd1260.out‚1.59375 disk.sda.out‚ 671.03515625 disk.sda2.out‚0" disk.nvme0n1.in‚34612.78515625' disk.nvme0n1.out‚270189.31884765625$ disk.dm-0.out‚270189.31884765625 network.sent‚ 2.7333406e+07 disk.sda2.in‚0$ disk.nvme0c0n1.in‚34612.78515625 disk.dm-0.in‚34612.78515625PË;䜧*‚(:‘ äªéÔ™âÝ® gpu.0.gpu‚47.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.33' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.666000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ây·„¨*‚(:ù äªéÔðÚÿ± disk.sda1.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚270566.9306640625& disk.nvme0n1.out‚270566.9306640625 disk.sda3.in‚0 disk.sda3.out‚ 671.58984375 cpu‚25.29587124015094 proc.cpu.threads‚17 network.sent‚ 2.7337492e+07% disk./.usageGB‚1173.2020263671875 disk.sda.out‚ 671.58984375! disk.nvme0n1.in‚ 34743.7109375 disk.dm-0.in‚ 34743.7109375$ memory_percent‚93.18134784698486) proc.memory.availableMB‚ 1117.16796875* proc.memory.percent‚18.003249168395996 disk.dm-1.in‚0 disk.dm-1.out‚ 671.58984375 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2949.65234375 network.recv‚ 3.793689e+06( disk./.usagePercent‚78.7345723715686 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.59765625 disk.sda.in‚0# disk.nvme0c0n1.in‚ 34743.7109375# disk.dm-0.out‚270566.9306640625ϯܔ4©*/ _wandbruntime‚11407  _runtime‚11407Xˆ;Ȫ* _runtime‚11407.594010029 _step‚876 & lr-AdamW/pg1‚3.151875460725362e-05 & lr-AdamW/pg2‚3.151875460725362e-05 trainer/global_step‚10699 $ _timestamp‚1.788499307953679e+09ÐRÜsÍ«*Ç $ _timestamp‚1.788499307953679e+09 _runtime‚11407.594010029 _step‚876 & lr-AdamW/pg1‚3.151875460725362e-05 & lr-AdamW/pg2‚3.151875460725362e-05 trainer/global_step‚10699ì’žûÀ4¬*/ _wandbruntime‚11407  _runtime‚11407^ñz (­*P [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10948408451994 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 trainer/global_step‚10699 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚747326362083840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚829922761665536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11457982910976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚829922737725440 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187098758 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚758314311062528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 424311859 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 237213101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚739194899736810 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚747326344080896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10987966981632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187098760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107261935 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 187098760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 $ _timestamp‚1.788499307954791e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚758315101710336 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 237213101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166742641 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚739195687424582n05{ (®*†P XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚747326344080896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10987966981632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187098760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107261935 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 187098760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 $ _timestamp‚1.788499307954791e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚758315101710336 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 237213101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166742641 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚739195687424582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10948408451994 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 trainer/global_step‚10699 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚747326362083840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚829922761665536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11457982910976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚829922737725440 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187098758 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚758314311062528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 424311859 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 237213101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚739194899736810 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664íÙc™4¯*/ _wandbruntime‚11408  _runtime‚11408è[ÐÙ °*ŒN PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚829998086068224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚739262755000428 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚747394044020224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 187115632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚758383806408704 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107271583 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚841457138809344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11458832574976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚728312574759936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 _step‚878 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚728312557720576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10949409591872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚758383806408704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 424373488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10989744385536 PBDeviceStatsMonitor.on_train_batch_e­Q&nd/inactive_split.all.allocated‚ 166757537 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚747394062023168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚739261967312448 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166756982 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚758383015747584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59485399 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚841456918643200 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 237234812 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59485952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 237234812 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚829998086068224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚739262755000428 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚747394044020224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 187115632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220166144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚758383806408704 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107271583 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚841457138809344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11458832574976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚728312574759936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 _step‚878 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚728312557720576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10949409591872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚758383806408704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 424373488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10989744385536 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166757537 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚747394062023168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚739261967312448 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166756982 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚758383015747584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59485399 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚841456918643200 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 237234812 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59485952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 237234812 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚833690154864640 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11510490179584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _runtime‚11456.511121067 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚761749559729152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11038991326720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚742548292109918 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 238325019 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 238302003 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚761750350376960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚761750350376960 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167490953 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚845200668984320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10999249685514 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167490398 % _timestamp‚1.7884993568706217e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187942460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 238325019 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187942458 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚750711341047296 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 187942458 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚833690178804736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _step‚881 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 426267479 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11038218681856 trainer/global_step‚10749 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚731549830112176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 109292µvÍÂ29312 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 187942460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11038991326720 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107744335 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59746616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992ZáH (À*‡P YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107744337 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚742549079797690 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚761749559729152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚750711359050240 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11038218681856 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚742548292109918 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 238325019 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 238302003 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚761750350376960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚761750350376960 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167490953 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚845200668984320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10999249685514 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167490398 % _timestamp‚1.7884993568706217e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187942460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 238325019 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187942458 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚750711341047296 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 187942458 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚833690178804736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 _step‚881 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 426267479 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11038218681856 trainer/global_step‚10749 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚731549830112176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 187942460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11038991326720 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107744335 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59746616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 426244461 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚750711359050240 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10998479037102 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59746063 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚845200444610560 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11510289745920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚731549813072816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚833690154864640 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11510490179584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _runtime‚11456.511121067 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚761749559729152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11038991326720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238323714 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚750779040986624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 426306090 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238323714 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚761818264414208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚742615359685556 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚551 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚845277075015680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚742616147373536 _step‚882 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 426283046 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238346756 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11039223427584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167505845 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59751311 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107753985 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 187959332 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59751860 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107753983 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 426306090 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238346756 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚750779058989568 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 426283046 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11511547868160 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 187959332 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚761819055075328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 187959334 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167505294 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚731615879508576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 " _timestamp‚1.7884993578273e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11039223427584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚549 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11039996085760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚11457.468093742 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚750779058989568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚731615896547936t|r‹'Ã*ŽN F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚731615879508576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 " _timestamp‚1.7884993578273e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11039223427584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚549 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11039996085760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚11457.468093742 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚750779058989568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚731615896547936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185740288 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11000250825600 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11039996085760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚833765503207424 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11511362127872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238323714 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚750779040986624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 426306090 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238323714 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚761818264414208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚742615359685556 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚551 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚845277075015680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚742616147373536 _step‚882 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 426283046 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238346756 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11039223427584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167505845 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59751311 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107753985 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 187959332 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59751860 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107753983 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 426306090 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238346756 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚750779058989568 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 426283046 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11511547868160 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 187959332 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚761819055075328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 187959334 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167505294òÐm4Ä*/ _wandbruntime‚11457  _runtime‚11457W8 ¼uÅ*Ü /loss/FRt1/detection_loss‚2.3517085022252537 .loss/FRt3/detection_loss‚2.867162302393338 1loss/SRqqt1/detection_loss‚2.0370282404124738 &loss/total_loss‚37.887352320445956 epoch‚8 _runtime‚11457.474510764 _step‚883 1loss/SRqqt4/detection_loss‚0.7588210641794527 /loss/FRt3/assignment_loss‚2.722119298474542 1loss/SRqqt3/detection_loss‚1.6674938622948736 .loss/FBt1/detection_loss‚1.992079569534822 /loss/FBt3/detection_loss‚2.2066994830965996 trainer/global_step‚10749 1loss/FBt2/assignment_loss‚0.29188998747849837 1loss/FBt3/assignment_loss‚0.14715082624343268 /loss/FRt4/detection_loss‚1.3551916170806166 1loss/SRqqt2/detection_loss‚2.3730177011178886 /loss/FBt4/detection_loss‚0.6480356027510621 % _timestamp‚1.7884993578343055e+09 0loss/FRt2/assignment_loss‚2.9690349902425495 2loss/SRqqt­—éPí3/assignment_loss‚1.3963089489572964 0loss/FBt4/assignment_loss‚0.1835141821370414 /loss/FBt2/detection_loss‚2.6537383794784546 0loss/FRt1/assignment_loss‚2.5000199908795566 2loss/SRqqt1/assignment_loss‚1.2724160087294876 2loss/SRqqt4/assignment_loss‚0.7358033509396288 /loss/FRt2/detection_loss‚2.6003424907698447 0loss/FRt4/assignment_loss‚0.7407190496959384 2loss/SRqqt2/assignment_loss‚1.1693380873488344 1loss/FBt1/assignment_loss‚0.24771878398446875»l4ÀgÆ*á 2loss/SRqqt1/assignment_loss‚1.2724160087294876 2loss/SRqqt4/assignment_loss‚0.7358033509396288 /loss/FRt2/detection_loss‚2.6003424907698447 0loss/FRt4/assignment_loss‚0.7407190496959384 2loss/SRqqt2/assignment_loss‚1.1693380873488344 1loss/FBt1/assignment_loss‚0.24771878398446875 /loss/FRt1/detection_loss‚2.3517085022252537 .loss/FRt3/detection_loss‚2.867162302393338 1loss/SRqqt1/detection_loss‚2.0370282404124738 &loss/total_loss‚37.887352320445956 epoch‚8 _runtime‚11457.474510764 _step‚883 1loss/SRqqt4/detection_loss‚0.7588210641794527 /loss/FRt3/assignment_loss‚2.722119298474542 1loss/SRqqt3/detection_loss‚1.6674938622948736 .loss/FBt1/detection_loss‚1.992079569534822 /loss/FBt3/detection_loss‚2.2066994830965996 trainer/global_step‚10749 1loss/FBt2/assignment_loss‚0.29188998747849837 1loss/FBt3/assignment_loss‚0.14715082624343268 /loss/FRt4/detection_loss‚1.3551916170806166 1loss/SRqqt2/detection_loss‚2.3730177011178886 /loss/FBt4/detection_loss‚0.6480356027510621 % _timestamp‚1.7884993578343055e+09 0loss/FRt2/assignment_loss‚2.9690349902425495 2loss/SRqqt3/assignment_loss‚1.3963089489572964 0loss/FBt4/assignment_loss‚0.1835141821370414 /loss/FBt2/detection_loss‚2.6537383794784546 0loss/FRt1/assignment_loss‚2.5000199908795566óâ¥l`’Ç*‚(:‡  «éÔ£ö§® gpu.0.gpu‚33.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.626' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.7252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éDL©€È*‚(:õ  «éÔ÷Žõ¯$ memory_percent‚92.66681671142578 network.sent‚ 2.7462568e+07% disk./.usageGB‚1172.1651077270508( disk.nvme0c0n1.out‚271667.9130859375 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.458629608154297 cpu‚25.29477574774836 network.recv‚ 3.812212e+06" disk.nvme0n1.in‚34928.69140625# disk.dm-0.out‚271667.9130859375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 673.4296875 disk.sda1.out‚0& disk.nvme0n1.out‚271667.9130859375 disk.dm-0.in‚34928.69140625 disk.sda3.out‚ 673.4296875 disk.drbd1260.out‚ 1.60546875& proc.memory.availableMB‚ 1201.46875! proc.memory.rssMB‚ 2860.421875 proc.cpu.threads‚17) disk./.usagePercent‚78.66498389159634$ disk.nvme0c0n1.in‚34928.69140625 disk.dm-1.out‚ 673.4296875 disk.sda.in‚0 disk.sda2.out‚0L+j`’É*‚(:‡ ¯«éÔ¡€Ì­ gpu.0.gpu‚28.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.868' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Så€Ê*‚(:ö ¯«éÔ‰¶ý® network.sent‚ 2.7570474e+07% disk./.usageGB‚1172.2503623962402$ disk.nvme0c0n1.in‚34928.69140625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚92.66963005065918( proc.memory.availableMB‚ 1201.0078125 network.recv‚ 3.817968e+06( disk.nvme0c0n1.out‚272059.8623046875& disk.nvme0n1.out‚272059.8623046875 disk.dm-0.in‚34928.69140625 disk.sda.in‚0 disk.sda.out‚ 673.8671875! proc.memory.rssMB‚ 2860.421875* proc.memory.percent‚17.458629608154297" disk.nvme0n1.in‚34928.69140625# disk.dm-0.out‚272059.8623046875 disk.sda3.in‚0 disk.sda3.out‚ 673.8671875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 cpu‚25.2945488059106) disk./.usagePercent‚78.67070540397907 disk.dm-1.out‚ 673.8671875 disk.drbd1260.out‚ 1.60546875 proc.cpu.threads‚17D’ˆ’Ë*‚(:‡ ¾«éÔÿ¸š® gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í=øÌ*‚(:ö ¾«éÔØñ¡° disk.sda2.out‚0) disk.nvme0c0n1.out‚272318.26611328125" disk.nvme0n1.in‚34928.69140625$ disk.dm-0.out‚272318.26611328125) proc.memory.availableMB‚ 1201.00390625! proc.memory.rssMB‚ 2860.421875 disk.drbd1260.out‚1.609375 disk.sda.out‚ 674.203125 disk.sda1.in‚0* proc.memory.percent‚17.458629608154297 cpu‚25.29426914298576 proc.cpu.threads‚17 network.sent‚ 2.7575552e+07 network.recv‚ 3.821455e+06) disk./.usagePercent‚78.66781175081955 disk.drbd1260.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚34928.69140625$ memory_percent‚92.66965389251709 disk.sda.in‚0 disk.sda1.out‚0' disk.nvme0n1.out‚272318.26611328125 disk.dm-0.in‚34928.69140625 disk.sda3.in‚0 disk.sda3.out‚ 674.203125 disk.dm-1.in‚0 disk.dm-1.out‚ 674.203125% disk./.usageGB‚1172.2072448730469Z)Øù4Í*/ _wandbruntime‚11503  _runtime‚11503 Ð{_ÊÎ*Ä _runtime‚11503.682189092 _step‚884 ' lr-AdamW/pg1‚2.7200597607433676e-05 ' lr-AdamW/pg2‚2.7200597607433676e-05 trainer/global_step‚10799 $ _timestamp‚1.788499404041814e+09ÔfH^ÏÏ*É _runtime‚11503.682189092 _step‚884 ' lr-AdamW/pg1‚2.7200597607433676e-05 ' lr-AdamW/pg2‚2.7200597607433676e-05 trainer/global_step‚10799 $ _timestamp‚1.788499404041814e+09ô!šI_4Ð*/ _wandbruntime‚11503  _runtime‚1150364J’(Ñ*€P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11088470382080 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 188786160 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 239390905 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚765185599043584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚745902472170798 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚734853134860816 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚765184808395776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚837457572003840 $ _timestamp‚1.788499404042936e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 239390905 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11089243026944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11089243026944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 _step‚885 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11049320270622 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992-V (Ò*…P \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚837457595943936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108226737 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 239413921 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11562894867968 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 188786158 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11563095301632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 239390905 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11089243026944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11089243026944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 _step‚885 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11049320270622 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11088470382080 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 188786160 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 239390905 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚765185599043584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚745902472170798 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚734853134860816 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚765184808395776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚837457572003840 $ _timestamp‚1.788499404042936e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚837532920346624 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 428215648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60018581 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚754164055955968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 188803032 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚765254303741952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 188803032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚849097086559744 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11049550762088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 428215648 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11090247785984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884994049083266e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚849096870587904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60018057 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pooÅ)dÁTl.peak‚ 1025507328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚837532944286720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11090247785984 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168254440 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 108236383 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 trainer/global_step‚10799 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚754164055955968 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239412616 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11564142273024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188803034 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚754164037953024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚765254303741952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚745969539746644 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11050321410708 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11089475127808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 239412616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 428238692 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 239435658 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 168254966 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚765253513080832°;`'Õ*‘N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 188803034 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 trainer/global_step‚10799 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚754164055955968 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239412616 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11564142273024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188803034 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚754164037953024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚765254303741952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚745969539746644 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11050321410708 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11089475127808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 239412616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 428238692 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 239435658 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 168254966 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚765253513080832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚734919218335936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚837532920346624 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 428215648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60018581 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚754164055955968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 188803032 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚765254303741952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 188803032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚849097086559744 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11049550762088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 428215648 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11090247785984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884994049083266e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚849096870587904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60018057 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚837532944286720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11090247785984 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168254440 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 108236383 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192031744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚768620057062400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚749255076856134 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚738156456648816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚757481334980096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11099390855730 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11138722082304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 % _timestamp‚1.7884994561546745e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 189629860 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11139494727168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11615471212032 _runtime‚11555.795124118 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚757481352983040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108709137 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11615669548544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚738156473688176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚852840460355072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168989815 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 G>DeviyœÕ×ôceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328hæt (æ*‡P YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108709137 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11615669548544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚738156473688176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚852840460355072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168989815 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60280117 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 240479807 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚563 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚852840682631680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240502823 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚749255864543906 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚889 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60280678 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11138722082304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚841224989143040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189629860 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11098620207318 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108709135 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚841225013083136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚768620057062400 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚768620847710208 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚757481352983040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 240479807 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚768620057062400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚749255076856134 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚738156456648816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚757481334980096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11099390855730 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11138722082304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 % _timestamp‚1.7884994561546745e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 189629860 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11139494727168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11615471212032 _runtime‚11555.795124118 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚757481352983040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ù98ɱ4ç*/ _wandbruntime‚11556  _runtime‚11556TÕu'è*ŒN MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚841300361425920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11140499486208 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 189646734 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11139726828032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚757549034919424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884994570077896e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚568 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108718785 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚757549052922368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189646732 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11099621347196 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚757549034919424 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169004207 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚768688761747456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11100391995816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 430171294 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚749322932119752 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚852916871153152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚738222540123936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 430148250 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 108718783 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240524560 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11139726828032 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 169004777 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240524560 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚768689552408576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚757549052922368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 189646734 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚749322144431772 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240501518 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 430148250Ú{¬I'é*‘N FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚757549052922368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189646732 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11099621347196 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚757549034919424 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169004207 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚768688761747456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11100391995816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 430171294 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚749322932119752 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚852916871153152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚738222540123936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 430148250 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 108718783 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240524560 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11139726828032 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 169004777 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240524560 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚768689552408576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚757549052922368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 189646734 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚749322144431772 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240501518 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 430148250 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚841300361425920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11140499486208 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 189646734 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11139726828032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚757549034919424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884994570077896e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚568 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108718785 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005ú¹Š¬4ê*/ _wandbruntime‚11556  _runtime‚11556RN¶?Xë*Ò _step‚891 /loss/FRt1/assignment_loss‚3.339075083795347 1loss/SRqqt2/assignment_loss‚1.716752459653295 .loss/FRt1/detection_loss‚2.464144674570937 /loss/FRt4/detection_loss‚1.7264766248189054 0loss/SRqqt3/detection_loss‚1.793332889517571 $ _timestamp‚1.788499457009993e+09 0loss/FRt4/assignment_loss‚1.0103570616470192 2loss/SRqqt4/assignment_loss‚0.9861757876506696 .loss/FBt1/detection_loss‚2.258280268737248 /loss/FBt2/detection_loss‚2.3326136469841003 -loss/FBt3/detection_loss‚2.42495910525322 /loss/FRt2/assignment_loss‚3.087775032967329 /loss/FRt3/assignment_loss‚2.781478735708421 2loss/FBt3/assignment_loss‚0.010766366036096287 .loss/FRt2/detection_loss‚2.448747977553012 1loss/FBt2/assignment_loss‚0.20589915857171945 0loss/FBt4/assignment_loss‚0.1464103425506604 /loss/FRt3/detection_loss‚2.3478412532037303 0loss/SRqqt2/detection_loss‚2.289038674189494 /loss/FBt4/detection_loss‚0.6055199131698293 %loss/total_loss‚39.81402453151426 _runtime‚11556.650144011 epoch‚8 trainer/global_step‚10849 2loss/SRqqt1/assignment_loss‚1.5493789817846846 2loss/SRqqt3/assignment_loss‚1.0503831546595122 1loss/FBt1/assignment_loss‚0.20304060700748647 1loss/SRqqt4/detection_loss‚0.8336770800459716 0loss/SRqqt1/detection_loss‚2.201899651437998õrÞì*× 1loss/FBt2/assignment_loss‚0.20589915857171945 0loss/FBt4/assignment_loss‚0.1464103425506604 /loss/FRt3/detection_loss‚2.3478412532037303 0loss/SRqqt2/detection_loss‚2.289038674189494 /loss/FBt4/detection_loss‚0.6055199131698293 %loss/total_loss‚39.81402453151426 _runtime‚11556.650144011 epoch‚8 trainer/global_step‚10849 2loss/SRqqt1/assignment_loss‚1.5493789817846846 2loss/SRqqt3/assignment_loss‚1.0503831546595122 1loss/FBt1/assignment_loss‚0.20304060700748647 1loss/SRqqt4/detection_loss‚0.8336770800459716 0loss/SRqqt1/detection_loss‚2.201899651437998 _step‚891 /loss/FRt1/assignment_loss‚3.339075083795347 1loss/SRqqt2/assignment_loss‚1.7167524596ïq¤‚53295 .loss/FRt1/detection_loss‚2.464144674570937 /loss/FRt4/detection_loss‚1.7264766248189054 0loss/SRqqt3/detection_loss‚1.793332889517571 $ _timestamp‚1.788499457009993e+09 0loss/FRt4/assignment_loss‚1.0103570616470192 2loss/SRqqt4/assignment_loss‚0.9861757876506696 .loss/FBt1/detection_loss‚2.258280268737248 /loss/FBt2/detection_loss‚2.3326136469841003 -loss/FBt3/detection_loss‚2.42495910525322 /loss/FRt2/assignment_loss‚3.087775032967329 /loss/FRt3/assignment_loss‚2.781478735708421 2loss/FBt3/assignment_loss‚0.010766366036096287 .loss/FRt2/detection_loss‚2.448747977553012ûÌ”/Î’í*‚(:‡ ‰¬éÔõ®ê¬ gpu.0.gpu‚38.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.9756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þbsguî*‚(:ê ‰¬éԽܼ± disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.6171875 disk.sda2.out‚0 disk.dm-0.in‚ 35080.625 disk.sda3.out‚ 676.44140625 disk.dm-1.out‚ 676.44140625 disk.sda.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚273519.33837890625 disk.nvme0n1.in‚ 35080.625$ disk.dm-0.out‚273519.33837890625 disk.sda3.in‚0! proc.memory.rssMB‚ 2860.421875 network.sent‚ 2.7706274e+07) disk./.usagePercent‚78.72551866512019 disk.sda.out‚ 676.44140625 disk.nvme0c0n1.in‚ 35080.625' disk.nvme0n1.out‚273519.33837890625% proc.memory.availableMB‚ 1200.9375* proc.memory.percent‚17.458629608154297 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚92.67005920410156 cpu‚25.29296958259543 network.recv‚ 3.844138e+06% disk./.usageGB‚1173.0671195983887É-AÍ‘ï*‚(:† ˜¬éÔΡ‡­ gpu.0.gpu‚23.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.025' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ÇÒ†ð*‚(:û ˜¬éÔ¹Èÿ± disk.sda.out‚ 676.79296875$ memory_percent‚92.67020225524902( proc.memory.availableMB‚ 1200.9140625* proc.memory.percent‚17.458629608154297 cpu‚25.29271919961847 network.recv‚ 3.850652e+06 disk.sda3.in‚0 disk.dm-1.out‚ 676.79296875 disk.sda.in‚0$ disk.nvme0c0n1.in‚35101.94921875) disk.nvme0c0n1.out‚273592.48583984375! proc.memory.rssMB‚ 2860.421875 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0' disk.nvme0n1.out‚273592.48583984375 network.sent‚ 2.7814927e+07) disk./.usagePercent‚78.72582254606333$ disk./.usageGB‚1173.071647644043 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.in‚35101.94921875 disk.dm-0.in‚ 35101.9609375$ disk.dm-0.out‚273592.48583984375 disk.sda3.out‚ 676.79296875 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.62109375yžN¤‘ñ*‚(:† §¬éÔâä—® gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚113.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”èdeŠò*‚(:ÿ §¬éÔ½†Ç° disk.sda.out‚ 677.24609375 disk.sda2.in‚0" disk.nvme0n1.in‚35239.06640625$ disk.dm-0.out‚273839.38818359375 cpu‚25.29635358826586 network.sent‚ 2.7820126e+07 disk.sda3.in‚0 disk.sda3.out‚ 677.24609375 disk.sda1.in‚0$ disk.nvme0c0n1.in‚35239.06640625' disk.nvme0n1.out‚273839.38818359375) proc.memory.availableMB‚ 1121.24609375) disk./.usagePercent‚78.73623386046582 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚273839.38818359375" proc.memory.rssMB‚ 2940.4296875 proc.cpu.threads‚17 network.recv‚ 3.854286e+06% disk./.usageGB‚1173.2267837524414 disk.sda1.out‚0 disk.dm-0.in‚35239.06640625$ memory_percent‚93.15645694732666* proc.memory.percent‚17.946958541870117 disk.dm-1.in‚0 disk.dm-1.out‚ 677.24609375 disk.drbd1260.out‚ 1.62109375ÜïÂx‘ó*‚(:† ¶¬éÔ·Ö£® gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚159.688' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.9376 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[4Ò‚ô*‚(:÷ ¶¬éÔ¸€Ü± disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚35239.06640625) disk.nvme0c0n1.out‚274119.99365234375$ memory_percent‚93.15798282623291) proc.memory.availableMB‚ 1120.99609375) disk./.usagePercent‚78.74668741611146% disk./.usageGB‚1173.3825492858887 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.946958541870117 cpu‚25.30298256784329 network.sent‚ 2.7825145e+07 network.recv‚ 3.857844e+06 disk.sda3.out‚ 677.8671875 disk.sda2.in‚0" proc.memory.rssMB‚ 2940.4296875 proc.cpu.threads‚17" disk.nvme0n1.in‚35239.06640625' disk.nvme0n1.out‚274119.99365234375 disk.dm-0.in‚35239.06640625$ disk.dm-0.out‚274119.99365234375 disk.dm-1.out‚ 677.8671875 disk.drbd1260.in‚0 disk.drbd1260.out‚1.625 disk.sda.out‚ 677.8671875T»°œõ*‚(:‘ ŬéÔ—˘® gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.674799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÐõÒ~ö*‚(:ó ŬéÔ±ªº¸ network.sent‚ 2.7833338e+07$ disk.nvme0c0n1.in‚35239.06640625" disk.nvme0n1.in‚35239.06640625 disk.drbd1260.out‚1.625 disk.sda.in‚0 disk.sda.out‚ 678.453125) disk.nvme0c0n1.out‚274440.02099609375' disk.nvme0n1.out‚274440.02099609375 disk.dm-0.in‚35239.06640625 disk.sda3.in‚0 disk.dm-1.out‚ 678.453125 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.out‚0) disk./.usagePercent‚78.72967315538584$ disk.dm-0.out‚274440.02099609375 disk.sda3.out‚ 678.453125 disk.dm-1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1121.24609375" proc.memory.rssMB‚ 2940.4296875 network.recv‚ 3.862895e+06% disk./.usageGB‚1173.1290245056152 disk.sda2.in‚0$ memory_percent‚93.15645694732666* proc.memory.percent‚17.946958541870117 cpu‚25.3098049406358 proc.cpu.threads‚17ÿh4š÷*‚(:… Ô¬éÔ¥Ó­ gpu.0.gpu‚55.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚177.53' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.506 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yïø*‚(:ô Ô¬éÔ»³º´ disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚274573.052734375" disk.dm-0.out‚274573.052734375 proc.cpu.threads‚17 disk.dm-1.out‚ 678.87890625 disk.drbd1260.in‚0 disk.sda.out‚ 678.87890625 disk.sda2.out‚0 disk.dm-0.in‚35239.06640625* proc.memory.percent‚17.946958541870117 network.sent‚ 2.7839208e+07) disk./.usagePercent‚78.72811048539684% disk./.usageGB‚1173.1057395935059 disk.sda1.out‚0$ disk.nvme0c0n1.in‚35239.06640625$ memory_percent‚93.15643310546875 network.recv‚ 3.867522e+06 disk.sda3.out‚ 678.87890625 disk.drbd1260.out‚ 1.62890625' disk.nvme0c0n1.out‚274573.052734375" disk.nvme0n1.in‚35239.06640625# proc.memory.availableMB‚1121.25" proc.memory.rssMB‚ 2940.4296875 cpu‚25.315654129351827äjÜ4ù*/ _wandbruntime‚11648  _runtime‚11648<¤+5m*ú*çT WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 170129948 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 375175680 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27846 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 598047232 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 241604722 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18215 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12155092992 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚761875683003904 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 190805456 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 432410178 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚761875683003904 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚11672324209152 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚753496644894890 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12155092992 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 598047232 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 11892004352 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚60721055 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚845938065230848 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111864 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 109408891 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚11193860174336 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 # _timestamp‚1.78849954880395e+09 _runtime‚11648.446868211 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚773070123222528 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚11672187027456 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 190805456 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚773070123222528 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 432391961 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1019674624 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18217 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27846 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18217 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 190805454 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚60720862 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚742343095797200 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 241586507«‹1 r*û*ìT LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882825 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚11154144209554 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚857610413380096 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚193 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578072504 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1019674624 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚18217 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27846 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18217 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 190805454 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚60720862 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚742343095797200 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 241586507 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 170129948 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 375175680 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚27846 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 598047232 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 241604722 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18215 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12155092992 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚761875683003904 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 190805456 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 432410178 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚761875683003904 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚11672324209152 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚753496644894890 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12155092992 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 598047232 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 11892004352 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚60721055 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚845938065230848 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111864 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 109408891 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚11193860174336 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 # _timestamp‚1.78849954880395e+09 _runtime‚11648.446868211 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚773070123222528 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚11672187027456 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 190805456 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚773070123222528 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 432391961 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚516 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚753539403895950 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 598047232 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚11194324922880 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚845985067316224 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚742385391637296 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚11194904967168 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚857657695152128 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 _runtime‚11649.875931004 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28115 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11129585664 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ºÈ'2 11672788957696 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1017634825 trainer/global_step‚10880 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580044288 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚773115457923072 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18215 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚197 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚857657856273920 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 190818428 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚11194324922880 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1046 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1005 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1005 _step‚893 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚516 % _timestamp‚1.7884995502322936e+09 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 375175680 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚516 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚845985043376128 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 10929229312 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚761921151003136 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 432413227 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892004352 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892004352 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚516 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚753539403895950 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 598047232 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚11194324922880 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚845985067316224 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚742385391637296 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚11194904967168 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚857657695152128 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 _runtime‚11649.875931004 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28115 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 10558147584 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 11129585664 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚11672788957696 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1017634825 trainer/global_step‚10880 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580044288 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚773115457923072 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18215 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚197 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚857657856273920 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 190818428 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚11194324922880 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1046 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1005 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1005 _step‚893 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚516 % _timestamp‚1.7884995502322936e+09 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 375175680 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚516 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚845985043376128 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 10929229312 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚761921151003136 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 432413227 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 B­$Ý0‚+ù¦ 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1846541315317154 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9934744238853455 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.44635725021362305 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,jet/accuracy_3_of_3‚0.028876807540655136 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 particle/accuracy_3_of_6‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 jet/accuracy_5_of_5‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4591503143310547 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988413453102112 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19724656641483307 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9906559586524963 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5495485663414001 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13509385287761688 *jet/accuracy_1_of_5‚0.9617447853088379 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454545319080353 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 jet/accuracy_1_of_6‚1 particle/accuracy_3_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.516565203666687 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10151372104883194 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3500398099422455 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39259254932403564 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5133333206176758 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5205733180046082 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.48528268933296204 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0606527104973793 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6309523582458496 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.010080644860863686 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5769522786140442 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27582329511642456 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 particle/accuracy_1_of_3‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5159281492233276 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5565552115440369 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026494737714529037 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21661144495010376 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2450539469718933 )jet/accuracy_1_of_1‚0.493527352809906 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.23111112415790558 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5635138154029846 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666865348816 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5214515328407288 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0021919880528002977 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.18390804529190063 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4814814329147339 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11964336782693863 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9883674383163452 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4734848737716675 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5844293832778931 *jet/accuracy_1_of_3‚0.8845273852348328 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3425666093826294 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.028754670172929764 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5609706044197083 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09282143414020538 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41607144474983215 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5334948897361755 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.585567057132721 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1361977458000183 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19848240911960602 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17656035721302032 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23732498288154602 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1132066547870636 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.07660818845033646 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3020799458026886 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15512733161449432 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3689520061016083 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6621527075767517 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_5_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26918885111808777 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25632181763648987 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5177874565124512 particle/accuracy_2_of_2‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9236111640930176 particle/accuracy_4_of_7‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968417882919312 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 (particle/accuracy‚0.8533141613006592 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.2003971834201“nÆì£b37e-05 particle/accuracy_1_of_2‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3958333432674408 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9911566376686096 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5427966117858887 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5632715821266174 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03837744891643524 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5008649826049805 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1019306480884552 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5016806125640869 particle/accuracy_4_of_5‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 % _timestamp‚1.7884995665116346e+09 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5320060849189758 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.15090088546276093 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.45679014921188354 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4591503441333771 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5401130318641663 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5646929740905762 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 epoch‚8 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870802760124207 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.49319925904273987 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3300594985485077 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3958333432674408 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5396857261657715 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21661144495010376 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03381141275167465 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25544866919517517 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6671702861785889 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5411364436149597 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.31593844294548035 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9901405572891235 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5013650059700012 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5270935893058777 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.549839437007904 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1888389140367508 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.914893627166748 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6305142045021057 *jet/accuracy_1_of_4‚0.9202508330345154 :"Purity/0FRt3SRqqt0FBt/event_purity‚0.03267974033951759 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2246326357126236 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5855098366737366 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03410778567194939 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.023740239441394806 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3661511540412903 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 :"Purity/0FRt3SRqqt*FBt/event_purity‚0.03267974033951759 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 5 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.99083012342453 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11975623667240143 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9916666746139526 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 _step‚894 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6501984000205994 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8153085708618164 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9832245707511902 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.025170069187879562 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.4487179219722748 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.18456126749515533 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_6‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04309524968266487 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 particle/accuracy_2_of_7‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22088466584682465 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03608042746782303 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2399805188179016 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.47391411662101746 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.990555465221405 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26151078939437866 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982810616493225 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.601367712020874 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6498016119003296 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5416666865348816 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5665438175201416 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5099454522132874 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5584173798561096 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 (jet/accuracy_1_of_2‚0.76421058177948 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ( Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.493036687374115 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20015248656272888 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_4_of_5‚0.01897081360220909 particle/accuracy_1_of_5‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2716049551963806 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1563786119222641 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666865348816 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 *jet/accuracy_2_of_5‚0.4549795091152191 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666865348816 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3009259104728699 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11520545184612274 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41358059644699097 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5456743836402893 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5346411466598511 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 &particle/f_score‚0.421908438205719 particle/accuracy_7_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4515014886856079 +jet/accuracy_3_of_4‚0.07804818451404572 jet/accuracy_4_of_7‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.06194968521595001 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19356030225753784 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *particle/specificity‚0.947745144367218 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.47655460238456726 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10823918133974075 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.09827585518360138 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23475340008735657 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8918128609657288 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4791666269302368 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30148065090179443 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 particle/accuracy_3_of_3‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9916690587997437 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9993895292282104 jet/accuracy_2_of_7‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26854002475738525 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6018344759941101 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_5‚0.08836404979228973 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14221569895744324 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5304597616195679 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01026438269764185 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985355794429779 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5571479797363281 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5888376832008362 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17445768415927887 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.706856369972229 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014813564717769623 -jet/accuracy_4_of_4‚0.0061450074426829815 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 jet/accuracy_7_of_7‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9947922229766846 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28225135803222656 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 trainer/global_step‚10880 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21377092599868774 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17409196496009827 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12007241696119308 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2630974054336548 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08464930951595306 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1113290786743164 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4683501422405243 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2172286957502365 jet/accuracy_3_of_7‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5792580246925354 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5704022645950317 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.409040242433548 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.990187406539917 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11923198401927948 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4625158905982971 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.006321839056909084 +particle/sensitivity‚0.3430629372596741 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.929894208908081 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4087301790714264 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2374967634677887 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.028830088675022125 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 jet/accuracy_5_of_6‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47941702604293823 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 +jet/accuracy_2_of_3‚0.26535096764564514 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.004051620606333017 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/event_purity‚0.01036866381764412 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9863478541374207 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.54155433177948 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36666664481163025 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1540403962135315 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05610119178891182 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31090086698532104 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9719535112380981 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5127781629562378 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15461227297782898 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9889556169509888 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 )loss/val_total_loss‚39.37701439132125 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2232142835855484 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 jet/accuracy_6_of_7‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 _runtime‚11666.154101931 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42384234070777893 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1031745970249176 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5618776679039001 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2320062220096588 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5358008146286011 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4074743986129761 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5349401831626892 particle/accuracy_2_of_5‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3102014362812042 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5016806125640869 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5478907227516174 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5433332920074463 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0031746032182127237 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.640350878238678 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18167103826999664 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5222868323326111 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.45354846119880676 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5459656119346619 +jet/accuracy_2_of_4‚0.40291324257850647 +jet/accuracy_3_of_6‚0.11538461595773697 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_6‚1*¼„ Oƒ+þ¦ & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.516565203666687 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10151372104883194 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3500398099422455 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39259254932403564 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5133333206176758 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5205733180046082 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.48528268933296204 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0606527104973793 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6309523582458496 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.010080644860863686 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5769522786140442 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27582329511642456 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 particle/accuracy_1_of_3‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5159281492233276 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5565552115440369 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026494737714529037 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21661144495010376 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2450539469718933 )jet/accuracy_1_of_1‚0.493527352809906 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.23111112415790558 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5635138154029846 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666865348816 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5214515328407288 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0021919880528002977 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.18390804529190063 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4814814329147339 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11964336782693863 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9883674383163452 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4734848737716675 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5844293832778931 *jet/accuracy_1_of_3‚0.8845273852348328 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3425666093826294 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.028754670172929764 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5609706044197083 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09282143414020538 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41607144474983215 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5334948897361755 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.585567057132721 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1361977458000183 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19848240911960602 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17656035721302032 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23732498288154602 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1132066547870636 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.07660818845033646 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚KÃä/6v0.3020799458026886 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15512733161449432 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3689520061016083 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6621527075767517 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_5_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26918885111808777 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25632181763648987 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5177874565124512 particle/accuracy_2_of_2‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9236111640930176 particle/accuracy_4_of_7‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968417882919312 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 (particle/accuracy‚0.8533141613006592 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3958333432674408 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9911566376686096 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5427966117858887 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5632715821266174 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03837744891643524 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5008649826049805 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1019306480884552 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5016806125640869 particle/accuracy_4_of_5‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 % _timestamp‚1.7884995665116346e+09 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5320060849189758 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.15090088546276093 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.45679014921188354 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4591503441333771 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5401130318641663 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5646929740905762 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 epoch‚8 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870802760124207 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.49319925904273987 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3300594985485077 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3958333432674408 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5396857261657715 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21661144495010376 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03381141275167465 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25544866919517517 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6671702861785889 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5411364436149597 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.31593844294548035 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9901405572891235 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5013650059700012 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5270935893058777 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.549839437007904 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1888389140367508 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.914893627166748 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6305142045021057 *jet/accuracy_1_of_4‚0.9202508330345154 :"Purity/0FRt3SRqqt0FBt/event_purity‚0.03267974033951759 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2246326357126236 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5855098366737366 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03410778567194939 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.023740239441394806 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3661511540412903 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 :"Purity/0FRt3SRqqt*FBt/event_purity‚0.03267974033951759 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 5 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.99083012342453 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11975623667240143 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9916666746139526 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 _step‚894 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6501984000205994 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8153085708618164 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9832245707511902 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.025170069187879562 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.4487179219722748 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.18456126749515533 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_6_of_6‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04309524968266487 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 particle/accuracy_2_of_7‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22088466584682465 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03608042746782303 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2399805188179016 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.47391411662101746 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.990555465221405 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26151078939437866 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982810616493225 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.601367712020874 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6498016119003296 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5416666865348816 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5665438175201416 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5099454522132874 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5584173798561096 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 (jet/accuracy_1_of_2‚0.76421058177948 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ( Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.493036687374115 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20015248656272888 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_4_of_5‚0.01897081360220909 particle/accuracy_1_of_5‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2716049551963806 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1563786119222641 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666865348816 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 *jet/accuracy_2_of_5‚0.4549795091152191 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666865348816 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3009259104728699 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11520545184612274 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41358059644699097 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5456743836402893 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5346411466598511 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 &particle/f_score‚0.421908438205719 particle/accuracy_7_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4515014886856079 +jet/accuracy_3_of_4‚0.07804818451404572 jet/accuracy_4_of_7‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.06194968521595001 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19356030225753784 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *particle/specificity‚0.947745144367218 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.47655460238456726 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10823918133974075 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.09827585518360138 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23475340008735657 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8918128609657288 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4791666269302368 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30148065090179443 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 particle/accuracy_3_of_3‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9916690587997437 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9993895292282104 jet/accuracy_2_of_7‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26854002475738525 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6018344759941101 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_5‚0.08836404979228973 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14221569895744324 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5304597616195679 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01026438269764185 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985355794429779 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5571479797363281 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5888376832008362 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17445768415927887 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.706856369972229 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014813564717769623 -jet/accuracy_4_of_4‚0.0061450074426829815 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 jet/accuracy_7_of_7‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9947922229766846 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28225135803222656 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 trainer/global_step‚10880 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21377092599868774 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17409196496009827 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.12007241696119308 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2630974054336548 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08464930951595306 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1113290786743164 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4683501422405243 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2172286957502365 jet/accuracy_3_of_7‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5792580246925354 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5704022645950317 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.409040242433548 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.990187406539917 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11923198401927948 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4625158905982971 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.006321839056909084 +particle/sensitivity‚0.3430629372596741 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.929894208908081 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4087301790714264 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2374967634677887 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.028830088675022125 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 jet/accuracy_5_of_6‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47941702604293823 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 +jet/accuracy_2_of_3‚0.26535096764564514 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.004051620606333017 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/event_purity‚0.01036866381764412 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9863478541374207 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.54155433177948 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36666664481163025 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1540403962135315 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05610119178891182 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31090086698532104 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9719535112380981 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5127781629562378 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15461227297782898 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9889556169509888 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 )loss/val_total_loss‚39.37701439132125 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2232142835855484 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 jet/accuracy_6_of_7‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 _runtime‚11666.154101931 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42384234070777893 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1031745970249176 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5618776679039001 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2320062220096588 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5358008146286011 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4074743986129761 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5349401831626892 particle/accuracy_2_of_5‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3102014362812042 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5016806125640869 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5478907227516174 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5433332920074463 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0031746032182127237 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.640350878238678 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18167103826999664 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5222868323326111 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.45354846119880676 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5459656119346619 +jet/accuracy_2_of_4‚0.40291324257850647 +jet/accuracy_3_of_6‚0.11538461595773697 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_6‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1846541315317154 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9934744238853455 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.44635725021362305 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,jet/accuracy_3_of_3‚0.028876807540655136 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 particle/accuracy_3_of_6‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 jet/accuracy_5_of_5‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4591503143310547 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988413453102112 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19724656641483307 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9906559586524963 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5495485663414001 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13509385287761688 *jet/accuracy_1_of_5‚0.9617447853088379 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454545319080353 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 jet/accuracy_1_of_6‚1 particle/accuracy_3_of_7‚1þŒ¦ Ñ„+‚(:„ ò¬éÔË©® gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.345' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.669 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 6ŸÈ‡…+‚(:ü ò¬éÔب´# proc.memory.rssMB‚ 3159.21484375) proc.memory.percent‚19.28231716156006 disk.sda1.in‚0 disk.sda1.out‚0# disk./.usageGB‚1173.51566696167 disk.sda3.in‚0 disk.sda.out‚ 679.67578125" disk.nvme0n1.in‚35363.28515625 disk.dm-0.in‚35363.28515625$ disk.dm-0.out‚274906.68310546875 network.recv‚ 3.877036e+06 disk.drbd1260.out‚ 1.6328125 disk.sda2.out‚0$ memory_percent‚91.08395576477051( proc.memory.availableMB‚ 1460.8046875 cpu‚25.326402149860634 proc.cpu.threads‚17 disk.sda3.out‚ 679.67578125 disk.dm-1.in‚0 disk.dm-1.out‚ 679.67578125 disk.drbd1260.in‚0 disk.sda.in‚0$ disk.nvme0c0n1.in‚35363.28515625) disk.nvme0c0n1.out‚274906.68310546875' disk.nvme0n1.out‚274906.68310546875 network.sent‚ 2.7950056e+07) disk./.usagePercent‚78.75562105502631 disk.sda2.in‚0;M g’†+‚(:‡ ­éÔÃñë­ gpu.0.gpu‚42.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.583' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãà–ÿ‡+‚(:û ­éÔ³Ùƒ² disk.sda3.out‚ 680.08203125 disk.dm-1.out‚ 680.08203125% disk./.usageGB‚1173.4911308288574$ disk.nvme0c0n1.in‚35363.28515625" disk.nvme0n1.in‚35363.28515625' disk.nvme0n1.out‚275704.96435546875 disk.sda.out‚ 680.08203125 disk.sda1.in‚0 network.recv‚ 3.88468e+06) disk./.usagePercent‚78.75397441456606 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚89.75427150726318) proc.memory.availableMB‚ 1678.66015625 disk.dm-1.in‚0 network.sent‚ 2.807518e+07 disk.dm-0.in‚35363.28515625$ disk.dm-0.out‚275704.96435546875 disk.sda3.in‚0 disk.drbd1260.out‚ 1.6328125 disk.sda1.out‚0) proc.memory.percent‚17.9472446441$Ï…§‡6504) disk.nvme0c0n1.out‚275704.96435546875" proc.memory.rssMB‚ 2940.4765625 cpu‚25.32571999324343 proc.cpu.threads‚17cÌÄZ4ˆ+/ _wandbruntime‚11689  _runtime‚116899¢gÂȉ+ & lr-AdamW/pg1‚2.318995446617167e-05 & lr-AdamW/pg2‚2.318995446617167e-05 trainer/global_step‚10899 $ _timestamp‚1.788499589566931e+09 _runtime‚11689.207323626 _step‚895@¶ÓHÍŠ+Ç _step‚895 & lr-AdamW/pg1‚2.318995446617167e-05 & lr-AdamW/pg2‚2.318995446617167e-05 trainer/global_step‚10899 $ _timestamp‚1.788499589566931e+09 _runtime‚11689.207323626ÿ—Þöß4‹+/ _wandbruntime‚11689  _runtime‚11689YùK(Œ+P $ _timestamp‚1.788499589568088e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 242093168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚774949876477440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191290796 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚763730833901056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 433406980 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚744124433402224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚11689.208576977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚774949085829632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11218269931520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191290794 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚755303091091172 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11177887040536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11218269931520 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109694275 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚847952053374976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11219042576384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚774949085829632 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚774949876477440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚847952029434880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60891682 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11697527224320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 _step‚896 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚755302303403400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11697721366528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664Á\³€ (+†P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚847952029434880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60891682 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11697527224320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 _step‚896 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚755302303403400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11697721366528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 $ _timestamp‚1.788499589568088e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 242093168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚774949876477440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191290796 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚763730833901056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 433406980 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚744124433402224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚11689.208576977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚774949085829632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11218269931520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191290794 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚755303091091172 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11177887040536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11218269931520 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109694275 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚847952053374976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11219042576384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194142208 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚774949085829632 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚774949876477440€ˆF4Ž+/ _wandbruntime‚11690  _runtime‚11690jãÄL'+‹N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60896742 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚859725937091584 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170600665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚744190482798624 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 191307670 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11219274677248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚775018581175808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11220047335424 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 433445591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109703923 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 433422547 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109703925 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 $ _timestamp‚1.788499590510418e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚848027401717760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚744190499837984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11698559313920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚775018581175808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚763798515837440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208808960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 trainer/global_step‚10899 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11220047335424 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191307668 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 433422547 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚763798515837440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992ô…ùNÞ+N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 191307670 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11219274677248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚775018581175808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11220047335424 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 433445591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109703923 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 433422547 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109703925 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 $ _timestamp‚1.788499590510418e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚848027401717760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚744190499837984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11698559313920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚775018581175808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚763798515837440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208808960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 trainer/global_step‚10899 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11220047335424 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191307668 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 433422547 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚763798515837440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60896742 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚859725937091584 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170600665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚744190482798624 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ú‹¯4‘+/ _wandbruntime‚11690  _runtime‚11690ãÕk]’+× .loss/FBt2/detection_loss‚2.508872240781784 /loss/FBt4/detection_loss‚0.5112688199963508 0loss/FRt3/assignment_loss‚1.9906196738111561 2loss/SRqqt4/assignment_loss‚0.7447392244109924 2loss/FBt1/assignment_loss‚0.002807734115046401 1loss/FBt2/assignment_loss‚0.16538244035143634 /loss/FRt1/detection_loss‚2.3893763973162723 2loss/SRqqt1/assignment_loss‚1.1120548486342159 _runtime‚11690.152560619 /loss/FRt1/assignment_loss‚2.822242252337627 1loss/FBt4/assignment_loss‚0.03161749468071596 .loss/FBt1/detection_loss‚3.203256130218506 /loss/FBt3/detection_loss‚2.8568602353334427 /loss/FRt2/assignment_loss‚2.496253796986171 2loss/SRqqt3/assignment_loss‚1.2690978404567685 /loss/FRt4/detection_loss‚1.5650590685637373 1loss/SRqqt1/detection_loss‚2.3121227249503136 1loss/SRqqt4/detection_loss‚0.8293036147753926 # _timestamp‚1.78849959051238e+09 1loss/FBt3/assignment_loss‚0.14093781492647275 /loss/FRt3/detection_loss‚2.9802342612167885 0loss/SRqqt2/detection_loss‚2.573839560702995 %loss/total_loss‚39.95594375016776 0loss/FRt4/assignment_loss‚1.1520784476347137 _step‚898 epoch‚9 trainer/global_step‚10899 2loss/SRqqt2/assignment_loss‚1.8686255801607061 .loss/FRt2/detection_loss‚2.513115426682957 0loss/SRqqt3/detection_loss‚1.916178121123203Nq·òb“+Ü epoch‚9 trainer/global_step‚10899 2loss/SRqqt2/assignment_loss‚1.8686255801607061 .loss/FRt2/detection_loss‚2.513115426682957 0loss/SRqqt3/detection_loss‚1.916178121123203 .loss/FBt2/detection_loss‚2.508872240781784 /loss/FBt4/detection_loss‚0.5112688199963508 0loss/FRt3/assignment_loss‚1.9906196738111561 2loss/SRqqt4/assignment_loss‚0.7447392244109924 2loss/FBt1/assignment_loss‚0.002807734115046401 1loss/FBt2/assignment_loss‚0.16538244035143634 /loss/FRt1/detection_loss‚2.3893763973162723 2loss/SRqqt1/assignment_loss‚1.1120548486342159 _runtime‚11690.152560619 /loss/FRt1/assignment_loss‚2.822242252337627 1loss/FBt4/assignment_loss‚0.03161749468071596 .loss/FBt1/detection_loss‚3.203256130218506 /loss/FBt3/detection_loss‚2.8568602353334427 /loss/FRt2/assignment_loss‚2.496253796986171 2loss/SRqqt3/assignment_loss‚1.2690978404567685 /loss/FRt4/detection_loss‚1.5650590685637373 1loss/SRqqt1/detection_loss‚2.3121227249503136 1loss/SRqqt4/detection_loss‚0.8293036147753926 # _timestamp‚1.78849959051238e+09 1loss/FBt3/assignment_loss‚0.14093781492647275 /loss/FRt3/detection_loss‚2.9802342612167885 0loss/SRqqt2/detection_loss‚2.573839560702995 %loss/total_loss‚39.95594375016776 0loss/FRt4/assignment_loss‚1.1520784476347137 _step‚898‚ ”q’”+‚(:‡ ­éÔ¶ˆ«¯ gpu.0.gpu‚38.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.558' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.1116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ!¥,ƒ•+‚(:ø ­éÔ„úü° disk.drbd1260.out‚1.640625% disk./.usageGB‚1173.5124130249023 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚35363.28515625$ disk.dm-0.out‚276591.98193359375 disk.sda3.out‚ 680.49609375" proc.memory.rssMB‚ 2940.4765625 cpu‚25.32543717897419( disk./.usagePercent‚78.7554026805996) disk.nvme0c0n1.out‚276591.98193359375" disk.nvme0n1.in‚35363.28515625 disk.sda3.in‚0$ memory_percent‚89.75467681884766& proc.memory.availableMB‚ 1678.59375 proc.cpu.threads‚17 disk.sda1.out‚0$ disk.nvme0c0n1.in‚35363.28515625' disk.nvme0n1.out‚276591.98193359375 disk.drbd1260.in‚0) proc.memory.percent‚17.94724464416504 network.sent‚ 2.8081045e+07 network.recv‚ 3.888966e+06 disk.sda.in‚0 disk.sda.out‚ 680.49609375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 680.49609375°›0–+‚(:„ Ÿ­éÔ˜«Ò¯ gpu.0.gpu‚38.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.55' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.11 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k÷N9€—+‚(:õ Ÿ­éÔ›€Ô³ network.recv‚ 3.895241e+06$ disk./.usageGB‚1173.637866973877 disk.sda3.in‚0 disk.dm-1.out‚ 680.9140625 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚35528.50390625) disk.nvme0c0n1.out‚276753.17724609375' disk.nvme0n1.out‚276753.17724609375) disk./.usagePercent‚78.76382200037816 disk.dm-1.in‚0 disk.sda.out‚ 680.9140625 disk.dm-0.in‚35528.50390625$ disk.dm-0.out‚276753.17724609375$ memory_percent‚89.75772857666016& proc.memory.availableMB‚ 1678.09375" proc.memory.rssMB‚ 2940.4765625 disk.sda3.out‚ 680.9140625 disk.sda.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.94724464416504 cpu‚25.32515972402956 network.sent‚ 2.8190929e+07 disk.drbd1260.out‚1.640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.in‚35528.50390625 proc.cpu.threads‚17_îÉ€œ˜+‚(:‘ ®­éÔäð¹­ gpu.0.gpu‚6.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.004' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.600799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä‹q¦†™+‚(:û ®­éÔÿ§«± disk.sda3.out‚ 681.33984375 disk.dm-1.out‚ 681.33984375 disk.sda.out‚ 681.33984375 disk.sda1.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚277081.2314453125 disk.dm-0.in‚35528.50390625 network.sent‚ 2.8194091e+07$ disk./.usageGB‚1174.241397857666$ memory_percent‚89.76008892059326" proc.memory.rssMB‚ 2940.4765625) proc.memory.percent‚17.94724464416504 network.recv‚ 3.899034e+06 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚35528.50390625& disk.nvme0n1.out‚277081.2314453125# disk.dm-0.out‚277081.2314453125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.64453125$ disk.nvme0c0n1.in‚35528.50390625) proc.memory.availableMB‚ 1677.70703125 cpu‚25.324884132564232 proc.cpu.threads‚17) disk./.usagePercent‚78.80432546438539ŸMŸ4š+/ _wandbruntime‚11738  _runtime‚11738ù·‹¢Ë›+Å % _timestamp‚1.7884996392842765e+09 _runtime‚11738.924821108 _step‚899 ' lr-AdamW/pg1‚2.1301005393586234e-05 ' lr-AdamW/pg2‚2.1301005393586234e-05 trainer/global_step‚10949z£õÐœ+Ê ' lr-AdamW/pg1‚2.1301005393586234e-05 ' lr-AdamW/pg2‚2.1301005393586234e-05 trainer/global_step‚10949 % _timestamp‚1.7884996392842765e+09 _runtime‚11738.924821108 _step‚899ƒ¥´ 4+/ _wandbruntime‚11738  _runtime‚11738bÕžê(ž+P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11269294276608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚778384334496256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚863469093619200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 243182070 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11268521631744 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 171330445 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚778385125144064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚767115812864512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚851719470514176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚767115830867456 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚747427755190224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _step‚900 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 192134494 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110176675 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 435316564 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚767115812864512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚758655695776508 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 243205086 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 243205086 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 192134496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 % _timestamp‚1.7884996392853656e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚778385125144064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚778384334496256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61153222 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚851719446574080 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61153768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11228728274056 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 trainer/global_step‚10949 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 _runtime‚11738.92599508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11269294276608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚863469315895808 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚747427738150864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110176677 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11227957625644 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016Ò=Ú›Ÿ+†P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚851719470514176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚767115830867456 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚747427755190224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 _step‚900 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 192134494 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110176675 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 435316564 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚767115812864512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚758655695776508 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 243205086 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 243205086 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 192134496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 % _timestamp‚1.7884996392853656e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚778385125144064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚778384334496256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61153222 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222276608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚851719446574080 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚Ü&_;÷ 489 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61153768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11228728274056 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 trainer/global_step‚10949 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 _runtime‚11738.92599508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11269294276608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚863469315895808 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚747427738150864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110176677 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11227957625644 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11269294276608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚778384334496256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚863469093619200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198336512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 243182070 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11268521631744 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 171330445 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚778385125144064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚767115812864512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0„ žo4 +/ _wandbruntime‚11739  _runtime‚11739oÙ'¡+‹N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171344852 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11270299035648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11750893695488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 192151368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 435378193 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 243226823 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11750691177984 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 192151370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚778453829842432 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚778453829842432 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110186323 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚863545712552448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚851794794916864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61159076 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚747493804586624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192151368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 243203781 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 435355149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚863545486094848 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11270299035648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚778453039181312 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11269526377472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 _runtime‚11739.811411027 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 $ _timestamp‚1.788499640170227e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 243203781 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 435355149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61158529 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11269526377472 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171345401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11229729414142 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚758722763352146 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚747493821625984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 243226823 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚767183530806784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312ŒVþ'¢+N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚851794818856960 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 435378193 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 192151370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚778453829842432 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚778453829842432 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110186323 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚863545712552448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚851794794916864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61159076 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚747493804586624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192151368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 243203781 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 435355149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚863545486094848 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11270299035648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚778453039181312 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11269526377472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 _runtime‚11739.811411027 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 $ _timestamp‚1.788499640170227e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 243203781 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 435355149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61158529 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11269526377472 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171345401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11229729414142 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚758722763352146 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚747493821625984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 243226823 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚767183530806784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171344852 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11270299035648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11750893695488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 192151368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 435378193 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 243226823 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11750691177984 BY4¬+/ _wandbruntime‚11788  _runtime‚11788^ºâË­+Å % _timestamp‚1.7884996887579336e+09 _runtime‚11788.398274331 _step‚903 ' lr-AdamW/pg1‚1.9490168257363054e-05 ' lr-AdamW/pg2‚1.9490168257363054e-05 trainer/global_step‚10999.n—GЮ+Ê _runtime‚11788.398274331 _step‚903 ' lr-AdamW/pg1‚1.9490168257363054e-05 ' lr-AdamW/pg2‚1.9490168257363054e-05 trainer/global_step‚10999 % _timestamp‚1.7884996887579336e+09‡D,k4¯+/ _wandbruntime‚11788  _runtime‚11788v)ð°+P GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 192978194 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 trainer/global_step‚10999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 192978194 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚781819583162880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 437249166 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚781820373810688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚855486887653376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚867288955048960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚855486863713280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 244270972 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 111295856641óp (±+†P JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚781819583162880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 437249166 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚781820373810688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚855486887653376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚867288955048960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚855486863713280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 244270972 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 192978194 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 trainer/global_step‚10999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 192978194 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚770568527773184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61420173 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚855562212056064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚762076155725254 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 192995070 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 192995068 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 192995070 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11802905633792 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 244315725 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110668723 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚750797126374624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚770568509770240 trainer/global_step‚10999 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 437310795 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11803106054144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚855562235996160 _runtime‚11789.24220377 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11279799999250 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚781888287847936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172088896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚781889078509056 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 437287751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚905 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192995068 % _timestamp‚1.7884996896013799e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 244315725 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11279029350630 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 437287751 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 437310795 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚867365117689856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚770568509770240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11320550735872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚762076943413234 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 244292683 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11320550735872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0)é ‹u´+N ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚750797143413984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11320550735872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚770568527773184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61420173 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚855562212056064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚762076155725254 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 192995070 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 192995068 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 192995070 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11802905633792 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 244315725 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110668723 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚750797126374624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ZFDeviceStatsMonitor.on_train_b®ãì¡atch_end/allocated_bytes.large_pool.freed‚770568509770240 trainer/global_step‚10999 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 437310795 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11803106054144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚855562235996160 _runtime‚11789.24220377 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11279799999250 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚781888287847936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172088896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚781889078509056 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 437287751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚905 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192995068 % _timestamp‚1.7884996896013799e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 244315725 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11279029350630 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 437287751 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 437310795 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚867365117689856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚770568509770240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11320550735872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚762076943413234 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 244292683 PaƒËÁ+Å _step‚907 ' lr-AdamW/pg1‚1.7757820433247147e-05 ' lr-AdamW/pg2‚1.7757820433247147e-05 trainer/global_step‚11049 % _timestamp‚1.7884997388703346e+09 _runtime‚11838.510823004µnþÐÂ+Ê ' lr-AdamW/pg1‚1.7757820433247147e-05 ' lr-AdamW/pg2‚1.7757820433247147e-05 trainer/global_step‚11049 % _timestamp‚1.7884997388703346e+09 _runtime‚11838.510823004 _step‚907‹s§›4Ã+/ _wandbruntime‚11838  _runtime‚11838Úë6d(Ä+P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 193821896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚871108510719488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11854205926912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 _step‚908 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 439181768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 _runtime‚11838.512416827 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚773885824800256 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 439204786 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚754034398766224 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11369797677056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 172817982 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11328869444272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 245359874 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 172817447 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚785255622477312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61676505 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚773885806797312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚773885806797312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11328098795860 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 $ _timestamp‚1.788499738871761e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11369797677056 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226470912 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11854003396096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚785254831829504 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 trainer/global_step‚11049 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11369025032192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 193821896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚859254304792576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚754034381726864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 245382890 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚765362480522724 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111141475 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11369025032192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 245359874 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 193821894 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚871108284248576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111141477 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚773885824800256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 193821894 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚785255622477312 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚785254831829504 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 trainer/global_step‚11049 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11369025032192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 193821896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚859254304792576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚754034381726864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 245382890 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚765362480522724 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111141475 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11369025032192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 245359874 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 193821894 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚871108284248576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111141477 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚773885824800256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 193821894 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚785255622477312 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 172817982 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11328869444272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 245359874 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 172817447 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚785255622477312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61676505 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚773885806797312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚773885806797312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11328098795860 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 $ _timestamp‚1.788499738871761e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11369797677056Œ&%a¿4Æ+/ _wandbruntime‚11839  _runtime‚11839Uóyõ'Ç+‹N YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11329099935738 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111151125 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788499740039997e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 245381585 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11370029777920 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚773953524739584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 245381585 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚773953506736640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11370802436096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193838770 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111151123 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172832387 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚785324327175680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚785324327175680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 172832929 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11855041741312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61681264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 193838768 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 439220353 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 439243397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61681804 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0èšÜ•'È+N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193838770 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111151123 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172832387 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚785324327175680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚785324327175680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 172832929 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11855041741312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61681264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 193838768 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 439220353 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 439243397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61681804 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11329099935738 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111151125 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788499740039997e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 245381585 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11370029777920 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚773953524739584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 245381585 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚773953506736640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11370802436096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÁvC4É+/ _wandbruntime‚11839  _runtime‚11839¥ìˆaÊ+Û 1loss/SRqqt2/detection_loss‚2.1799094311652647 %loss/total_loss‚39.18269523374842 /loss/FRt1/detection_loss‚2.5672892709507753 1loss/SRqqt4/detection_loss‚0.7736172549548033 0loss/FRt2/assignment_loss‚2.9434945661088694 0loss/FRt4/assignment_loss‚0.7968401129429157 2loss/SRqqt1/assignment_loss‚1.7391018279912798 2loss/SRqqt4/assignment_loss‚0.7783720072196878 1loss/FBt1/assignment_loss‚0.07913477240673833 trainer/global_step‚11049 % _timestamp‚1.7884997400428226e+09 /loss/FRt1/assignment_loss‚3.246767934999968 2loss/SRqqt3/assignment_loss‚1.1858848862349987 1loss/SRqqt3/detection_loss‚1.7837153472006322 epoch‚9 _runtime‚11839.683109494 2loss/FBt4/assignment_loss‚0.053219863785083955 /loss/FBt1/detection_loss‚2.3246374766031903 _step‚910 /loss/FRt3/assignment_loss‚1.838959841892637 2loss/SRqqt2/assignment_loss‚1.4088416516540512 /loss/FBt2/detection_loss‚2.2955684711535773 /loss/FBt4/detection_loss‚0.3616340087001147 1loss/FBt3/assignment_loss‚0.18764228401505534 .loss/FRt2/detection_loss‚2.849104210765765 /loss/FRt3/detection_loss‚2.4358858568915007 /loss/FRt4/detection_loss‚1.2455127850174903 /loss/FBt3/detection_loss‚2.9328214824199677 0loss/FBt2/assignment_loss‚0.2387433968733982 0loss/SRqqt1/detection_loss‚2.935996491800655i„ÅfË+à % _timestamp‚1.7884997400428226e+09 /loss/FRt1/assignment_loss‚3.246767934999968 2loss/SRqqt3/assignment_loss‚1.1858848862349987 1loss/SRqqt3/detection_loss‚1.7837153472006322 epoch‚9 _runtime‚11839.683109494 2loss/FBt4/assignment_loss‚0.053219863785083955 /loss/FBt1/detection_loss‚2.3246374766031903 _step‚910 /loss/FRt3/assignment_loss‚1.838959841892637 2loss/SRqqt2/assignment_loss‚1.4088416516540512 /loss/FBt2/detection_loss‚2.2955684711535773 /loss/FBt4/detection_loss‚0.3616340087001147 1loss/FBt3/assignment_loss‚0.18764228401505534 .loss/FRt2/detection_loss‚2.849104210765765 /loss/FRt3/detection_loss‚2.4358858568915007 /loss/FRt4/detection_loss‚1.2455127850174903 /loss/FBt3/detection_loss‚2.9328214824199677 0loss/FBt2/assignment_loss‚0.2387433968733982 0loss/SRqqt1/detection_loss‚2.935996491800655 1loss/SRqqt2/detection_loss‚2.1799094311652647 %loss/total_loss‚39.18269523374842 /loss/FRt1/detection_loss‚2.5672892709507753 1loss/SRqqt4/detection_loss‚0.7736172549548033 0loss/FRt2/assignment_loss‚2.9434945661088694 0loss/FRt4/assignment_loss‚0.7968401129429157 2loss/SRqqt1/assignment_loss‚1.7391018279912798 2loss/SRqqt4/assignment_loss‚0.7783720072196878 1loss/FBt1/assignment_loss‚0.07913477240673833 trainer/global_step‚11049Ž§Þ Ì+‚(:’ ¦®éԼ蓭 gpu.0.gpu‚47.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.242' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.848399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ…ÃÝ~Í+‚(:ó ¦®éÔÖʯ®" memory_percent‚89.776611328125) proc.memory.percent‚17.94724464416504 cpu‚25.322677051034848 disk.sda3.in‚0 disk.dm-1.out‚ 684.80859375 disk.drbd1260.out‚ 1.6640625 disk.sda.in‚0$ disk.nvme0c0n1.in‚35797.29296875 disk.dm-0.in‚35797.29296875 proc.memory.availableMB‚1675" proc.memory.rssMB‚ 2940.4765625% disk./.usageGB‚1174.9379272460938 disk.dm-1.in‚0$ disk.dm-0.out‚279636.50146484375 proc.cpu.threads‚17) disk./.usagePercent‚78.85107013606985 disk.sda3.out‚ 684.80859375 disk.drbd1260.in‚0 disk.sda.out‚ 684.80859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0' disk.nvme0n1.out‚279636.50146484375 network.sent‚ 2.8444411e+07 network.recv‚ 3.936538e+06 disk.sda2.out‚0) disk.nvme0c0n1.out‚279636.50146484375" disk.nvme0n1.in‚35797.29296875UêÍÎ+‚(:’ µ®éÔ¨Þ„­ gpu.0.gpu‚44.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.565' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.312999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ!ª|Ï+‚(:ñ µ®éÔµòê¯) disk./.usagePercent‚78.85264944654857 disk.sda.in‚0 disk.sda.out‚ 685.328125 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚35797.29296875" disk.nvme0n1.in‚35797.29296875 disk.dm-0.in‚35797.29296875 network.sent‚ 2.8555432e+07% disk./.usageGB‚1174.9614601135254 disk.dm-1.in‚0 disk.dm-1.out‚ 685.328125 disk.drbd1260.out‚ 1.6640625 disk.sda1.in‚0' disk.nvme0n1.out‚280036.93896484375$ disk.dm-0.out‚280036.93896484375 disk.sda3.in‚0$ memory_percent‚89.77813720703125) proc.memory.percent‚17.94724464416504 cpu‚25.32241194388213 proc.cpu.threads‚17 network.recv‚ 3.943418e+06 disk.sda3.out‚ 685.328125 disk.drbd1260.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚280036.93896484375# proc.memory.availableMB‚1674.75" proc.memory.rssMB‚ 2940.4765625kc&Ð+‚(:’ Ä®éÔ¨Òß­ gpu.0.gpu‚21.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.069' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.213800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥nÍf‚Ñ+‚(:÷ Ä®éÔœÛÚ¯" proc.memory.rssMB‚ 2940.4765625 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.out‚ 685.640625 disk.sda.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚280341.53173828125$ memory_percent‚89.77665901184082( proc.memory.availableMB‚ 1674.9921875) proc.memory.percent‚17.94724464416504 network.sent‚ 2.8559582e+07) disk./.usagePercent‚78.85264842251843% disk./.usageGB‚1174.9614448547363 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.66796875) disk.nvme0c0n1.out‚280341.53173828125 cpu‚25.32215436952983 network.recv‚ 3.947893e+06 disk.sda1.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚35797.29296875 disk.dm-0.in‚35797.29296875$ disk.dm-0.out‚280341.53173828125 disk.sda3.out‚ 685.640625 disk.sda.out‚ 685.640625 disk.sda1.out‚0" disk.nvme0n1.in‚35797.29296875ž¿4Ò+/ _wandbruntime‚11888  _runtime‚11888L·ŽLÉÓ+à & lr-AdamW/pg1‚1.610432293992855e-05 & lr-AdamW/pg2‚1.610432293992855e-05 trainer/global_step‚11099 % _timestamp‚1.7884997890342455e+09 _runtime‚11888.674581474 _step‚911ÿLB¾ÎÔ+È & lr-AdamW/pg1‚1.610432293992855e-05 & lr-AdamW/pg2‚1.610432293992855e-05 trainer/global_step‚11099 % _timestamp‚1.7884997890342455e+09 _runtime‚11888.674581474 _step‚911ª×µ¤4Õ+/ _wandbruntime‚11888  _runtime‚11888ë…¸fu Ö+‚P `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚777270821766656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 194665596 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 246471792 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚788690080496128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11420049377280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚768715872895832 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 246471792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 441114370 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 194665596 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚768716660583604 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111623877 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.=| >“small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚788690871143936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚788690871143936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 246448776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 194665594 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚863021721931776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11905958221312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 173563432 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 % _timestamp‚1.7884997890353153e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚788690080496128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 173562912 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚874927876392448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 173563432 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 % _timestamp‚1.7884997890353153e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚788690080496128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 173562912 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚874927876392448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚788690080496128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11420049377280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚768715872895832 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 246471792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 441114370 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 194665596 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚768716660583604 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111623877 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚788690871143936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚788690871143936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 246448776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489"Ö ÿ4Ø+/ _wandbruntime‚11889  _runtime‚11889cBý'Ù+‹N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚768783728159450 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61944819 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 194682468 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 441152955 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194682470 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚788759575842304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61944292 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚875004269074944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 $ _timestamp‚1.788499790053112e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚863097070274560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11379170520846 trainer/global_step‚11099 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚757403786989984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 246470487 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _step‚913 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚777338503703040 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173578344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚757403769950624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚768782940471470 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚777338521705984 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111633525 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11907198800384 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11379941169466 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173577815 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111633523 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚875004042617344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 441175999 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚788759575842304 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11421054136320 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚777338521705984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11420281478144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 441152955 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 194682468 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11906996282880 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 194682470 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005.€Ü\øÚ+N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61944819 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 194682468 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 441152955 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194682470 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚788759575842304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61944292 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚875004269074944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 $ _timestamp‚1.788499790053112e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚863097070274560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11379170520846 trainer/global_step‚11099 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚757403786989984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 246470487 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _step‚913 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚777338503703040 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173578344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚757403769950624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚768782940471470 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚777338521705984 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111633525 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11907198800384 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11379941169466 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173577815 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111633523 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚875004042617344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 441175999 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚788759575842304 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11421054136320 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚777338521705984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11420281478144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 441152955 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 194682468 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11906996282880 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 194682470 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226457600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚768783728159450 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328‘ÖÄ?z4Û+/ _wandbruntime‚11889  _runtime‚11889Mù·vYÜ+Ó 1loss/SRqqt3/assignment_loss‚1.354235873762834 1loss/FBt2/assignment_loss‚0.06684825296365814 /loss/FRt2/detection_loss‚2.4757611812219174 1loss/SRqqt2/detection_loss‚2.3471075654029847 /loss/FBt4/detection_loss‚0.5830270058708266 _step‚914 0loss/FRt4/assignment_loss‚1.2913515132571023 0loss/FBt4/assignment_loss‚0.1035742737344421 epoch‚9 1loss/SRqqt4/assignment_loss‚0.938552542817374 1loss/FBt3/assignment_loss‚0.10322753793132412 .loss/FBt1/detection_loss‚3.356726798144254 %loss/total_loss‚40.07090389246696 $ _timestamp‚1.788499790056005e+09 _runtime‚11889.696260663 /loss/FRt2/assignment_loss‚3.042230667386736 1loss/FBt1/assignment_loss‚0.11927467085535103 /loss/FBt3/detection_loss‚2.1395205042578955 /loss/FRt3/assignment_loss‚2.626837091199283 1loss/SRqqt1/assignment_loss‚1.470728077063406 trainer/global_step‚11099 2loss/SRqqt2/assignment_loss‚1.4318761727639606 .loss/FRt1/detection_loss‚2.614217251730271 /loss/FRt3/detection_loss‚2.7081986175015054 /loss/FRt4/detection_loss‚1.4105716659170058 0loss/SRqqt1/detection_loss‚2.437300197504185 0loss/SRqqt4/detection_loss‚0.713638732412263 0loss/FRt1/assignment_loss‚2.4246189151054773 1loss/SRqqt3/detection_loss‚1.8979946445610563 .loss/FBt2/detection_loss‚2.413484139101846Зb^Ý+Ø 0loss/SRqqt1/detection_loss‚2.437300197504185 0loss/SRqqt4/detection_loss‚0.713638732412263 0loss/FRt1/assignment_loss‚2.4246189151054773 1loss/SRqqt3/detection_loss‚1.8979946445610563 .loss/FBt2/detection_loss‚2.413484139101846 1loss/SRqqt3/assignment_loss‚1.354235873762834 1loss/FBt2/assignment_loss‚0.06684825296365814 /loss/FRt2/detection_loss‚2.4757611812219174 1loss/SRqqt2/detection_loss‚2.3471075654029847 /loss/FBt4/detection_loss‚0.5830270058708266 _step‚914 0loss/FRt4/assignment_loss‚1.2913515132571023 0loss/FBt4/assignment_loss‚0.1035742737344421 epoch‚9 1loss/SRqqt4/assignment_loss‚0.938552542817374 1loss/FBt3/assignment_loss‚0.10322753793132412 .loss/FBt1/detection_loss‚3.356726798144254 %loss/total_loss‚40.07090389246696 $ _timestamp‚1.788499790056005e+09 _runtime‚11889.696260663 /loss/FRt2/assignment_loss‚3.042230667386736 1loss/FBt1/assignment_loss‚0.11927467085535103 /loss/FBt3/detection_loss‚2.1395205042578955 /loss/FRt3/assignment_loss‚2.626837091199283 1loss/SRqqt1/assignment_loss‚1.470728077063406 trainer/global_step‚11099 2loss/SRqqt2/assignment_loss‚1.4318761727639606 .loss/FRt1/detection_loss‚2.614217251730271 /loss/FRt3/detection_loss‚2.7081986175015054 /loss/FRt4/detection_loss‚1.4105716659170058’ÕqDÿÞ+‚(:’ Ó®éÔ‚È•® gpu.0.gpu‚43.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚165.837' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.167399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04³{—Šß+‚(:ÿ Ó®éÔز° disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚35797.29296875) disk.nvme0c0n1.out‚280693.07861328125' disk.nvme0n1.out‚280693.07861328125 network.sent‚ 2.8563669e+07 disk.sda3.in‚0 disk.drbd1260.out‚ 1.66796875" disk.nvme0n1.in‚35797.29296875$ disk.dm-0.out‚280693.07861328125$ memory_percent‚89.76891040802002) proc.memory.availableMB‚ 1676.26171875 cpu‚25.321873204183177 disk.dm-0.in‚35797.29296875" proc.memory.rssMB‚ 2940.4765625) proc.memory.percent‚17.94724464416504 proc.cpu.threads‚17 network.recv‚ 3.950562e+06) disk./.usagePercent‚78.80307896370117% disk./.usageGB‚1174.2228240966797 disk.dm-1.out‚ 686.00390625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 686.00390625 disk.sda1.in‚0 disk.sda3.out‚ 686.00390625é#ñŽà+‚(:’ â®éÔäøö­ gpu.0.gpu‚50.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.369' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.273799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 îÿ¢„á+‚(:ù â®éÔÁ‘¦± disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚280992.95166015625$ disk.dm-0.out‚280992.95166015625" proc.memory.rssMB‚ 2940.4765625) proc.memory.percent‚17.94724464416504 disk.sda3.in‚0 disk.sda3.out‚ 686.5546875 disk.dm-1.out‚ 686.5546875 disk.drbd1260.out‚1.671875 disk.sda1.out‚0$ disk.nvme0c0n1.in‚35944.58984375 disk.dm-0.in‚35944.58984375 proc.cpu.threads‚17 network.recv‚ 3.956406e+06) disk./.usagePercent‚78.80304465869158 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 686.5546875) disk.nvme0c0n1.out‚280992.95166015625" disk.nvme0n1.in‚35944.58984375$ memory_percent‚89.76895809173584$ disk./.usageGB‚1174.222312927246 disk.drbd1260.in‚0) proc.memory.availableMB‚ 1676.25390625 cpu‚25.321591524279167 network.sent‚ 2.8671717e+07 disk.sda1.in‚0Ð'ºVâ+‚(:… ñ®éÔâí’° gpu.0.gpu‚41.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|QÙ†ã+‚(:û ñ®éÔ¤õµ²' disk.nvme0n1.out‚281188.76611328125 disk.dm-0.in‚35944.58984375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚35944.58984375) proc.memory.availableMB‚ 1676.50390625 network.recv‚ 3.961292e+06 disk.dm-1.out‚ 687.17578125 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.80325176878678$ disk./.usageGB‚1174.225399017334 disk.sda3.in‚0 disk.sda3.out‚ 687.17578125 disk.dm-1.in‚0" proc.memory.rssMB‚ 2940.4765625) proc.memory.percent‚17.94724464416504 cpu‚25.321309718886884" disk.nvme0n1.in‚35944.58984375$ disk.dm-0.out‚281188.76611328125 disk.drbd1260.in‚0 disk.drbd1260.out‚1.671875 disk.sda.in‚0 disk.sda.out‚ 687.17578125) disk.nvme0c0n1.out‚281188.76611328125$ memory_percent‚89.76743221282959 network.sent‚ 2.867783e+07ãfCu4ä+/ _wandbruntime‚11937  _runtime‚11937j‡VzËå+Å trainer/global_step‚11149 % _timestamp‚1.7884998374765694e+09 _runtime‚11937.116939853 _step‚915 ' lr-AdamW/pg1‚1.4530020363806546e-05 ' lr-AdamW/pg2‚1.4530020363806546e-05ß'£èÐæ+Ê % _timestamp‚1.7884998374765694e+09 _runtime‚11937.116939853 _step‚915 ' lr-AdamW/pg1‚1.4530020363806546e-05 ' lr-AdamW/pg2‚1.4530020363806546e-05 trainer/global_step‚11149“tqð4ç+/ _wandbruntime‚11937  _runtime‚11937Eà4©(è+‚P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195509294 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚760641042342224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚792125329162752 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _step‚916 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚780655800730112 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174308750 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174308206 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11428239966076 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11469528432640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚878747544163328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 % _timestamp‚1.7884998374776418e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112106275 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 195509296 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚866789139070976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 247560694 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11470301077504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11429010614488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚780655818733056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206725120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 247537678 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚866789115130880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 443069990 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11958198367232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚792126119810560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62202473 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚792125329162752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11470301077504œEŸÐ (é+‡P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112106277 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230665216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚772069265268940 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195509294 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚780655818733056 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62201931 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 trainer/global_step‚11149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11470301077504 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195509294 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚760641042342224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚792125329162752 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _step‚916 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚780655800730112 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174308750 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174308206 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11428239966076 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11469528432640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚878747544163328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 % _timestamp‚1.7884998374776418e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112106275 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 195509296 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚866789139070976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 247560694 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11470301077504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11429010614488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚780655818733056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206725120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 247537678 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚866789115130880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 443069990 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11958198367232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚792126119810560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62202473 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚792125329162752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚780723500669440 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11149 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 443085557 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11470533178368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 247559389 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚792194824508928 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 195526168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247559389 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11470533178368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 % _timestamp‚1.7884998385544991e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11959240680960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174323667 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11471305836544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 443108601 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚780723518672384 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚780723518672384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 195526170 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174323122 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680x^9Z'ì+‘N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 195526168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247559389 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11470533178368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 % _timestamp‚1.7884998385544991e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11959240680960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174323667 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11471305836544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 443108601 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚780723518672384 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚780723518672384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 195526170 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174323122 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚760707108777984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 443085557 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚780723500669440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 195526168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚760707091738624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11430011754574 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11471305836544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚780723500669440 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11149 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 443085557 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11470533178368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 247559389 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚792194824508928 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 196352994 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11520552777728 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11519780132864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 248626580 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112588677 _step‚920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11479081199596 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚784040815699456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 248649596 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175054961 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62465749 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚784040815699456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚795560577829376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚870556556210176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 196352996 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208822272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11520552777728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11478310551184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 196352994 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 445002592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚882567272744448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 248626580 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175054424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚795560577829376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚870556556210176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 196352996 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208822272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11520552777728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11478310551184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 196352994 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 445002592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚882567272744448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 248626580 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175054424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 196352994 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11520552777728 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11519780132864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 248626580 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112588677 _step‚920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11479081199596 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚784040815699456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 248649596 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175054961 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62465749 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚784040815699456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175069221 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚795630073175552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚784108515638784 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196369870 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 445018159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚784108515638784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112598323 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 196369868 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚870631904552960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚784108497635840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11480082339682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚764010430565984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248671333 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 445041203 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62471442 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 196369870 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 248648291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚882643655873024 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11520784878592 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 175069767 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚921 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 248671333 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚764010413526624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 _runtime‚11989.817123917 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11521557536768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 445018159 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12011751320064 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884998901762211e+098]%p€,‘N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚795630073175552 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚882643431512576 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12011550899712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚921 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 248671333 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚764010413526624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 _runtime‚11989.817123917 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11521557536768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 445018159 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12011751320064 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884998901762211e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚775490512905666 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚775489725217686 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175069221 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚795630073175552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚784108515638784 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196369870 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 445018159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚784108515638784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112598323 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 196369868 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚870631904552960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚784108497635840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11480082339682 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚764010430565984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.cuñþñð§rrent‚ 787687980 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248671333 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 445041203 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62471442 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 196369870 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 248648291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚882643655873024 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11520784878592 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 175069767 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 FÜ ’†,‚(:‡ ˯éÔð™Ñ¯ gpu.0.gpu‚26.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=-ª‡,‚(:ô ˯éÔÇÃÀ°% disk./.usageGB‚1174.2906455993652 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 689.7421875 disk.sda1.out‚0( disk.nvme0c0n1.out‚283211.8056640625! disk.nvme0n1.in‚ 36080.3515625 disk.dm-1.in‚0 disk.dm-1.out‚ 689.7421875 disk.drbd1260.out‚ 1.68359375# disk.nvme0c0n1.in‚ 36080.3515625 disk.dm-0.in‚ 36080.3515625" proc.memory.rssMB‚ 2940.4765625) proc.memory.percent‚17.94724464416504 proc.cpu.threads‚17 network.sent‚ 2.891727e+07 network.recv‚ 3.990135e+06) disk./.usagePercent‚78.80763052165241 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚93.17946434020996( proc.memory.availableMB‚ 1117.4765625 cpu‚25.319671468907874 disk.sda3.out‚ 689.7421875 disk.drbd1260.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚283211.8056640625# disk.dm-0.out‚283211.8056640625œ“Ȉ,‚(:’ Ú¯éÔ™±Š­ gpu.0.gpu‚48.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.480400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OŽ£‰,‚(:ô Ú¯éÔžœÍ® disk.sda2.out‚0) disk./.usagePercent‚78.80920548000306 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚283598.22900390625 disk.sda3.out‚ 690.3671875" proc.memory.rssMB‚ 2940.4765625 network.sent‚ 2.8922352e+07 network.recv‚ 3.993673e+06 disk.sda.out‚ 690.3671875# disk.nvme0c0n1.in‚ 36080.3515625' disk.nvme0n1.out‚283598.22900390625) proc.memory.availableMB‚ 1117.73046875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 690.3671875 disk.drbd1260.in‚0 disk.sda2.in‚0$ disk.dm-0.out‚283598.22900390625) proc.memory.percent‚17.94724464416504 cpu‚25.319428199629435% disk./.usageGB‚1174.3141136169434 disk.drbd1260.out‚1.6875! disk.nvme0n1.in‚ 36080.3515625 disk.dm-0.in‚ 36080.3515625 disk.sda3.in‚0# memory_percent‚93.1779146194458Ô6žÿ4Š,/ _wandbruntime‚12039  _runtime‚12039Ù"ºË‹,Å ' lr-AdamW/pg1‚1.1620295719867456e-05 ' lr-AdamW/pg2‚1.1620295719867456e-05 trainer/global_step‚11249 % _timestamp‚1.7884999402847762e+09 _runtime‚12039.925435241 _step‚923ñÂsƒÐŒ,Ê ' lr-AdamW/pg1‚1.1620295719867456e-05 ' lr-AdamW/pg2‚1.1620295719867456e-05 trainer/global_step‚11249 % _timestamp‚1.7884999402847762e+09 _runtime‚12039.925435241 _step‚923›îæµä4,/ _wandbruntime‚12039  _runtime‚12039¾’#œ(Ž,‚P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 197196694 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12063041045504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787425812665856 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚874323949409280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11529151784704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 197196694 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62729921 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚798995826496000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884999402865915e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚798995826496000 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 249738498 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 249715482 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 _step‚924¯ÀŒë (,‡P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884999402865915e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚798995826496000 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 249738498 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 249715482 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 _step‚924 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 197196694 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12063041045504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787425812665856 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚874323949409280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11529151784704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 197196694 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62729921 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚798995826496000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539œV|•C4,/ _wandbruntime‚12041  _runtime‚12041a? Z‘,‹N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 175815947 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚778843117590794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚767313752353984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11571036578816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚799065321842176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 102550732N¿:1·"8 _runtime‚12041.014242216 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 197213570 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62734683 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚799064531181056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11571809236992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175815406 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12064090588160 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787493512605184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚886463412280320 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11529382276170 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 446950761 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11571036578816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11530152924790 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚787493494602240 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113080723 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62735222 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 446973805 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚787493512605184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚787493494602240 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚778843905278774 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚886463198405632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 _step‚925 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113080723 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62735222 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 446973805 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚787493512605184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚787493494602240 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚778843905278774 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚886463198405632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 _step‚925 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11571036578816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚799065321842176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _runtime‚12041.014242216 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 197213570 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62734683 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚799064531181056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11571809236992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175815406 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12064090588160 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787493512605184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚886463412280320 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11529382276170 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 446950761 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11571036578816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11530152924790 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚787493494602240 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚782129442388264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 250827400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚802431865810432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12115327036416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 448844778 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚790810791629312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113553475 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚802431075162624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚790810809632256 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11621056178176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 250804384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11579222369812 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 448867796 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 198040396 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚770550990666864 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 176546382 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62993445 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208822272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _step‚928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 250804384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚878091390488576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 198040396 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _runtime‚12088.898974861 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 250827400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11620283533312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12115118214144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11621056178176 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool‘õäê‹.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚782130230076036 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62992907 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 $ _timestamp‚1.788499989258385e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 198040394 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚770551007706224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚802431075162624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚790810791629312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚802431865810432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚782129442388264 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 250827400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚802431865810432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12115327036416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 448844778 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚790810791629312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113553475 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚802431075162624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚790810809632256 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11621056178176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 250804384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11579222369812 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 448867796 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 198040396 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚770550990666864 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 176546382 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62993445 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208822272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 _step‚928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 250804384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚878091390488576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 198040396 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 _runtime‚12088.898974861 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 250827400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11620283533312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12115118214144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11621056178176 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚782130230076036 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62992907 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 $ _timestamp‚1.788499989258385e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 198040394 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚770551007706224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚802431075162624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚790810791629312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚802431865810432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176561289 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 250849137 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 250849137 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚802499779847680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚878166738831360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 448906407 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 448883363 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11621288279040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113563123 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 _runtime‚12089.75756988 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12116366180352 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚770617074141984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 176561824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚890282884845568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚782197297651882 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198057268 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62998699 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11299 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11622060937216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11621288279040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚878166714891264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113563125 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 % _timestamp‚1.7884999901166356e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚802500570508800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 448883363 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11579452861278 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚790878491568640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚890283105011712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚790878509571584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 448906407 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 250826095 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 198057268 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11580223509898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚802500570508800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _step‚929 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125]$+"¤,N FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 250849137 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 250849137 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚802499779847680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚878166738831360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 448906407 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 448883363 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11621288279040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113563123 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 196226048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 _runtime‚12089.75756988 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12116366180352 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚770617074141984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 176561824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚890282884845568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚782197297651882 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198057268 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62998699 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11299 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11622060937216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11621288279040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚878166714891264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113563125 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 % _timestamp‚1.7884999901166356e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚802500570508800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 448883363 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11579452861278 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚790878491568640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚890283105011712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚790878509571584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 448906407 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 250826095 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 198057268 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11580223509898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚802500570508800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 _step‚929 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176561289 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046¡N4ÐK4¥,/ _wandbruntime‚12089  _runtime‚12089r@i^¦,Ø 2loss/FBt2/assignment_loss‚0.024741845407918715 /loss/FRt4/detection_loss‚1.0738360317690032 .loss/FBt3/detection_loss‚2.589801049232483 /loss/FRt3/assignment_loss‚2.555843309776203 /loss/FRt4/assignment_loss‚1.100230936966245 .loss/FRt2/detection_loss‚2.642219670643996 1loss/SRqqt1/detection_loss‚2.2727264195680617 1loss/SRqqt4/detection_loss‚0.8112424238723643 /loss/FBt2/detection_loss‚3.4464385168892995 % _timestamp‚1.7884999901190064e+09 /loss/FRt3/detection_loss‚2.9815489665882007 /loss/FBt4/detection_loss‚0.4483341592418797 0loss/FRt1/assignment_loss‚2.6786721878581576 1loss/FBt3/assignment_loss‚0.03250380270473225 .loss/FBt1/detection_loss‚2.658200098408593 0loss/FRt2/assignment_loss‚2.3318691831082106 1loss/SRqqt1/assignment_loss‚1.153533356015881 2loss/SRqqt2/assignment_loss‚1.2005125152645633 1loss/FBt4/assignment_loss‚0.12188946047969547 1loss/SRqqt3/detection_loss‚2.0287292137359962 _runtime‚12089.75917696 _step‚930 0loss/SRqqt2/detection_loss‚2.313228437677026 epoch‚9 0loss/FBt1/assignment_loss‚0.3389818097109835 trainer/global_step‚11299 2loss/SRqqt3/assignment_loss‚1.5995344090729187 2loss/SRqqt4/assignment_loss‚0.8340602980956356 /loss/FRt1/detection_loss‚2.4260861261023416 &loss/total_loss‚39.664764228190386ñ\-^c§,Ý 2loss/FBt2/assignment_loss‚0.024741845407918715 /loss/FRt4/detection_loss‚1.0738360317690032 .loss/FBt3/detection_loss‚2.589801049232483 /loss/FRt3/assignment_loss‚2.555843309776203 /loss/FRt4/assignment_loss‚1.100230936966245 .loss/FRt2/detection_loss‚2.642219670643996 1loss/SRqqt1/detection_loss‚2.2727264195680617 1loss/SRqqt4/detection_loss‚0.8112424238723643 /loss/FBt2/detection_loss‚3.4464385168892995 % _timestamp‚1.7884999901190064e+09 /loss/FRt3/detection_loss‚2.9815489665882007 /loss/FBt4/detection_loss‚0.4483341592418797 0loss/FRt1/assignment_loss‚2.6786721878581576 1loss/FBt3/assignment_loss‚0.03250380270473225 .loss/FBt1/detection_loss‚2.658200098408593 0loss/FRt2/assignment_loss‚2.3318691831082106 1loss/SRqqt1/assignment_loss‚1.153533356015881 2loss/SRqqt2/assignment_loss‚1.2005125152645633 1loss/FBt4/assignment_loss‚0.12188946047969547 1loss/SRqqt3/detection_loss‚2.0287292137359962 _runtime‚12089.75917696 _step‚930 0loss/SRqqt2/detection_loss‚2.313228437677026 epoch‚9 0loss/FBt1/assignment_loss‚0.3389818097109835 trainer/global_step‚11299 2loss/SRqqt3/assignment_loss‚1.5995344090729187 2loss/SRqqt4/assignment_loss‚0.8340602980956356 /loss/FRt1/detection_loss‚2.4260861261023416 &loss/total_loss‚39.664764228190386¢ÐæÉ’¨,‚(:‡ –°éÔŸóí¯ gpu.0.gpu‚30.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.144' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2288 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—Œö©,‚(:ö –°éÔи½°% disk.nvme0n1.out‚285046.689453125 disk.sda1.out‚0$ memory_percent‚93.15481185913086' disk.nvme0c0n1.out‚285046.689453125 disk.dm-0.in‚36275.25390625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2940.4765625) proc.memory.percent‚17.94724464416504" disk.dm-0.out‚285046.689453125 network.recv‚ 4.012059e+06 disk.sda2.in‚0$ disk.nvme0c0n1.in‚36275.25390625 disk.sda.out‚ 692.39453125 disk.sda1.in‚0' proc.memory.availableMB‚ 1121.515625 cpu‚25.318358827780525 network.sent‚ 2.9047633e+07 disk.sda3.in‚0 disk.dm-1.out‚ 692.39453125 disk.drbd1260.out‚ 1.6953125 proc.cpu.threads‚17 disk.sda3.out‚ 692.39453125) disk./.usagePercent‚78.74682002801418% disk./.usageGB‚1173.3845252990723 disk.sda2.out‚0" disk.nvme0n1.in‚36275.25390625©½±Îœª,‚(:‘ ¥°éÔŒúÈ® gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.128' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.22559999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kJu\n«,‚(:ã ¥°éÔêÀ—¯) proc.memory.percent‚17.94724464416504 proc.cpu.threads‚17' disk.nvme0c0n1.out‚285400.236328125 disk.drbd1260.out‚ 1.6953125( proc.memory.availableMB‚ 1122.3515625" proc.memory.rssMB‚ 2940.4765625 cpu‚25.31805459495356 network.sent‚ 2.915663e+07 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 36408.2421875! disk.nvme0n1.in‚ 36408.2421875% disk.nvme0n1.out‚285400.236328125 disk.sda3.in‚0 disk.sda3.out‚692.875) disk./.usagePercent‚78.75101496747024 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 36408.2421875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚692.875 network.recv‚ 4.019016e+06% disk./.usageGB‚1173.4470329284668" disk.dm-0.out‚285400.236328125 disk.dm-1.out‚692.875$ memory_percent‚93.14970970153809•ý®_’¬,‚(:‡ ´°éÔ²¹ù­ gpu.0.gpu‚50.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.623' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ünW…­,‚(:ú ´°éÔ–ÈÁ°" proc.memory.rssMB‚ 2940.4765625) proc.memory.percent‚17.94724464416504# disk.nvme0c0n1.in‚ 36408.2421875! disk.nvme0n1.in‚ 36408.2421875 disk.sda3.out‚ 693.33203125 disk.dm-1.out‚ 693.33203125 cpu‚25.31778080309837 proc.cpu.threads‚17% disk./.usageGB‚1173.5954818725586) disk.nvme0c0n1.out‚285751.80419921875 disk.dm-0.in‚ 36408.2421875 disk.drbd1260.in‚0 disk.sda.in‚0$ memory_percent‚93.14830303192139 network.sent‚ 2.9164445e+07( disk./.usagePercent‚78.7609775006652 disk.sda2.out‚0' disk.nvme0n1.out‚285751.80419921875$ disk.dm-0.out‚285751.80419921875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.69921875 network.recv‚ 4.023578e+06 disk.sda2.in‚0 disk.sda.out‚ 693.33203125 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1122.58203125á…¾4®,/ _wandbruntime‚12134  _runtime‚12134aJàɯ,à trainer/global_step‚11349 % _timestamp‚1.7885000351728215e+09 _runtime‚12134.813221684 _step‚931 & lr-AdamW/pg1‚9.031071904079096e-06 & lr-AdamW/pg2‚9.031071904079096e-06¡Gœqΰ,È trainer/global_step‚11349 % _timestamp‚1.7885000351728215e+09 _runtime‚12134.813221684 _step‚931 & lr-AdamW/pg1‚9.031071904079096e-06 & lr-AdamW/pg2‚9.031071904079096e-06£ý×#ï4±,/ _wandbruntime‚12134  _runtime‚12134 í®²(²,€P @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚ 12134.8146414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 450777380 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 198884094 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11349 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚881858807627776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11671307878400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 198884096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885000351740823e+09 _step‚932 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚794195806598656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11670535233536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 450800398 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177294029 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63258708 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 251916302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198884094 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177294585 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚785482834761372 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12167654603264 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 198884096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114035875 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63258154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚881858783687680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚805866323829248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚805866323829248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114035877 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚773854312454864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11628522306508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12167454169600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚794195788595712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046ögª (³,…P UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚785482834761372 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12167654603264 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 198884096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114035875 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63258154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚881858783687680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚805866323829248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚805866323829248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114035877 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚773854312454864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11628522306508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12167454169600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚794195788595712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚ 12134.8146414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 450777380 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 198884094 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11349 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚881858807627776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11671307878400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 198884096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885000351740823e+09 _step‚932 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚794195806598656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11670535233536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 450800398 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177294029 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63258708 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 251916302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198884094 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 177294585¤á‹å’´,‚(:‡ ðéÔþ칯 gpu.0.gpu‚28.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0604 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Dzš†µ,‚(:û ðéÔآᱠnetwork.recv‚ 4.027161e+06 disk.sda.out‚ 693.54296875) disk.nvme0c0n1.out‚286079.06201171875! disk.nvme0n1.in‚ 36408.2421875 disk.dm-0.in‚ 36408.2421875$ disk.dm-0.out‚286079.06201171875$ memory_percent‚93.14682483673096) proc.memory.availableMB‚ 1122.82421875) disk./.usagePercent‚78.77782510048603 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2940.4765625 cpu‚25.317487503364248$ disk./.usageGB‚1173.846523284912 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 36408.2421875 disk.sda3.in‚0 disk.sda3.out‚ 693.54296875 proc.cpu.threads‚17 network.sent‚ 2.9180672e+07 disk.drbd1260.out‚ 1.69921875' disk.nvme0n1.out‚286079.06201171875 disk.dm-1.in‚0 disk.dm-1.out‚ 693.54296875) proc.memory.percent‚17.94724464416504Ï´Õ4¶,/ _wandbruntime‚12135  _runtime‚12135§J1©Ú·,ŠN RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11629523446386 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 177309565 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63263507 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 trainer/global_step‚11349 _step‚933 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚805935819175424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 450815965 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 198900970 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11672312637440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚805935819175424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚794263488535040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198900970 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11671539979264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177309030 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198900968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚894102657868288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚785549902337010 _runtime‚12135.821143265 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 450839009 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12168707383808 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 198900968 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚894102863354368 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 198900970 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11672312637440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚805935819175424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚794263488535040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198900970 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11671539979264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177309030 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198900968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚894102657868288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚785549902337010 _runtime‚12135.821143265 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 450839009 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12168707383808 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 198900968 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚894102863354368 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11629523446386 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 177309565 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176¥(ç¼à4¹,/ _wandbruntime‚12135  _runtime‚12135Ýú¤^º,Ø /loss/FRt4/detection_loss‚1.4509347416460514 /loss/FRt1/assignment_loss‚2.755949935247732 /loss/FRt3/detection_loss‚2.4959323837568888 1loss/SRqqt2/detection_loss‚2.2742247069254518 _runtime‚12135.822739043 _step‚934 0loss/FRt4/assignment_loss‚1.3657989657443503 /loss/FBt3/detection_loss‚1.9499048855569627 /loss/FBt4/detection_loss‚0.5683378634459683 2loss/SRqqt1/assignment_loss‚1.2381487642956335 1loss/FBt1/assignment_loss‚0.27489416332290756 1loss/FBt4/assignment_loss‚0.06667849592472626 0loss/FRt2/assignment_loss‚2.5863869653807745 0loss/FRt3/assignment_loss‚1.9081743955612183 2loss/SRqqt3/assignment_loss‚1.2118726744739847 2loss/SRqqt4/assignment_loss‚0.8029343512426648 0loss/FBt2/assignment_loss‚0.1985976076159659 /loss/SRqqt1/detection_loss‚2.05637827030448 trainer/global_step‚11349 1loss/SRqqt2/assignment_loss‚1.531210362445563 /loss/FBt2/detection_loss‚2.2941130499045053 %loss/total_loss‚36.22247632562095 epoch‚9 2loss/FBt3/assignment_loss‚0.061160073704130805 /loss/FRt1/detection_loss‚2.1968775725988454 0loss/SRqqt3/detection_loss‚1.573513962328434 -loss/FBt1/detection_loss‚2.32973974943161 % _timestamp‚1.7885000361825483e+09 /loss/FRt2/detection_loss‚2.1209124671093176 1loss/SRqqt4/detection_loss‚0.9097999176527891ê_¼c»,Ý /loss/FRt1/assignment_loss‚2.755949935247732 /loss/FRt3/detection_loss‚2.4959323837568888 1loss/SRqqt2/detection_loss‚2.2742247069254518 _runtime‚12135.822739043 _step‚934 0loss/FRt4/assignment_loss‚1.3657989657443503 /loss/FBt3/detection_loss‚1.9499048855569627 /loss/FBt4/detection_loss‚0.5683378634459683 2loss/SRqqt1/assignment_loss‚1.2381487642956335 1loss/FBt1/assignment_loss‚0.27489416332290756 1loss/FBt4/assignment_loss‚0.06667849592472626 0loss/FRt2/assignment_loss‚2.5863869653807745 0loss/FRt3/assignment_loss‚1.9081743955612183 2loss/SRqqt3/assignment_loss‚1.2118726744739847 2loss/SRqqt4/assignment_loss‚0.8029343512426648 0loss/FBt2/assignment_loss‚0.1985976076159659 /loss/SRqqt1/detection_loss‚2.05637827030448 trainer/global_step‚11349 1loss/SRqqt2/assignment_loss‚1.531210362445563 /loss/FBt2/detection_loss‚2.2941130499045053 %loss/total_loss‚36.22247632562095 epoch‚9 2loss/FBt3/assignment_loss‚0.061160073704130805 /loss/FRt1/detection_loss‚2.1968775725988454 0loss/SRqqt3/detection_loss‚1.573513962328434 -loss/FBt1/detection_loss‚2.32973974943161 % _timestamp‚1.7885000361825483e+09 /loss/FRt2/detection_loss‚2.1209124671093176 1loss/SRqqt4/detection_loss‚0.9097999176527891 /loss/FRt4/detection_loss‚1.4509347416460514¦×Ù½¼,‚(:’ Ò°éÔ²†”® gpu.0.gpu‚37.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.497' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.299400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c%Ÿ½,‚(:ö Ò°éÔÇü“² network.sent‚ 2.9264236e+07# disk.nvme0c0n1.in‚ 36408.2421875! disk.nvme0n1.in‚ 36408.2421875 disk.sda3.in‚0$ memory_percent‚93.14696788787842) proc.memory.availableMB‚ 1122.80078125) proc.memory.percent‚17.94724464416504 proc.cpu.threads‚17$ disk./.usageGB‚1173.845531463623 disk.sda2.out‚0) disk.nvme0c0n1.out‚286262.54833984375 disk.drbd1260.out‚1.703125 disk.sda.in‚0 network.recv‚ 4.032591e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 694.2421875 disk.sda.out‚ 694.2421875" proc.memory.rssMB‚ 2940.4765625 cpu‚25.317198848216687) disk./.usagePercent‚78.77775853852714 disk.sda1.in‚0' disk.nvme0n1.out‚286262.54833984375 disk.dm-0.in‚ 36408.2421875$ disk.dm-0.out‚286262.54833984375 disk.dm-1.in‚0 disk.dm-1.out‚ 694.2421875 disk.drbd1260.in‚0ö1ä’¾,‚(:‡ á°éÔ¾‘„¯ gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.626' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.7252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡Z3ì¿,‚(:ö á°éÔðóû² cpu‚25.316929133726283 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚286469.81396484375) proc.memory.availableMB‚ 1122.80078125) proc.memory.percent‚17.94724464416504 disk.drbd1260.out‚1.703125) disk./.usagePercent‚78.77696875528399% disk./.usageGB‚1173.8337631225586$ disk.dm-0.out‚286469.81396484375 disk.sda3.out‚ 694.734375 disk.dm-1.out‚ 694.734375 disk.sda.in‚0 disk.sda1.in‚0 network.recv‚ 4.039379e+06' disk.nvme0n1.out‚286469.81396484375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 694.734375 disk.sda2.in‚0$ disk.nvme0c0n1.in‚36546.04296875$ memory_percent‚93.14696788787842 network.sent‚ 2.936288e+07" disk.nvme0n1.in‚36546.04296875 disk.dm-0.in‚36546.04296875 disk.sda2.out‚0" proc.memory.rssMB‚ 2940.47656259냑À,‚(:† ð°éÔóØ»¯ gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.315' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.063 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hổÁ,‚(:þ ð°éÔŠ†â² proc.cpu.threads‚17) disk./.usagePercent‚78.77618230013881% disk./.usageGB‚1173.8220443725586 disk.dm-1.in‚0 disk.dm-1.out‚ 695.26171875 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 695.26171875 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 1.70703125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚36546.04296875" disk.nvme0n1.in‚36546.04296875' disk.nvme0n1.out‚286941.61474609375 network.sent‚ 2.9368896e+07) disk.nvme0c0n1.out‚286941.61474609375$ disk.dm-0.out‚286941.61474609375) proc.memory.percent‚17.94724464416504 network.recv‚ 4.044131e+06 disk.sda3.out‚ 695.26171875 disk.dm-0.in‚36546.04296875$ memory_percent‚93.14541816711426( proc.memory.availableMB‚ 1123.0546875" proc.memory.rssMB‚ 2940.4765625 cpu‚25.3166442514652352L4Â,/ _wandbruntime‚12182  _runtime‚12182z¼ÈŽÅÃ,¿ # _timestamp‚1.78850008274511e+09 _runtime‚12182.385505126 _step‚935 % lr-AdamW/pg1‚7.85733274596406e-06 % lr-AdamW/pg2‚7.85733274596406e-06 trainer/global_step‚11399,ß\ãÊÄ,Ä % lr-AdamW/pg1‚7.85733274596406e-06 % lr-AdamW/pg2‚7.85733274596406e-06 trainer/global_step‚11399 # _timestamp‚1.78850008274511e+09 _runtime‚12182.385505126 _step‚935§ªâgd4Å,/ _wandbruntime‚12182  _runtime‚12182Ö¢‚Æ,P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11679363540028 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚885626200826880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 199727796 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 452709982 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 199727794 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚797580785562112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12219999641600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11399 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚809301572495872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 _runtime‚12182.386974575 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11721559578624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚897846004229120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚788836227134480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚809302363143680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 199727794 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 252982188 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚897846224408576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 253005204 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114518275 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 199727794 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 252982188 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚897846224408576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 253005204 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114518275 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11679363540028 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚885626200826880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 199727796 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 452709982 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 199727794 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚797580785562112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12219999641600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11399 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚809301572495872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 _runtime‚12182.386974575 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11721559578624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚897846004229120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚788836227134480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚809302363143680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 253026941 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 253003899 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 452771611 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 199744670 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11680364680114 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚897922624904192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚897922415223808 _step‚937 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 199744670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚788904082398098 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 452748567 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63530269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 452771611 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12221051794432 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12220866054144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚788903294710118 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 253003899 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11721791679488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚809371067842048 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚809370277180928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 452748567 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11722564337664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11721791679488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185740288 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178058192 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114527925 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚809371067842048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚777223700678624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚797648503504384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11679594031494 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 trainer/global_step‚11399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚885701573109760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚777223717717984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚885701549169664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 199744668 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 253026941 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚809371067842048 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚809370277180928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 452748567 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11722564337664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11721791679488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 185740288 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178058192 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114527925 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚809371067842048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚777223700678624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚797648503504384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11679594031494 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209680384 trainer/global_step‚11399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚885701573109760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚777223717717984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚885701549169664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 199744668 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 253026941 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 253026941 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 253003899 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 452771611 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 199744670 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11680364680114 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚897922624904192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚897922415223808 _step‚937 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 199744670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚788904082398098 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 452748567 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63530269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 452771611 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12221051794432 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12220866054144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚788903294710118 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 253003899 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11721791679488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F™å…Ñ,‚(:ú Ž±éÔâÊŸ¸ network.recv‚ 4.053875e+06% disk./.usageGB‚1174.1789512634277 disk.dm-1.out‚ 696.15234375 disk.sda1.in‚0" proc.memory.rssMB‚ 2940.4765625 cpu‚25.316214821576857 network.sent‚ 2.9483649e+07) disk./.usagePercent‚78.80013462104986 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.7109375 disk.sda.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚36546.04296875( disk.nvme0c0n1.out‚287622.1806640625" disk.nvme0n1.in‚36546.04296875& disk.nvme0n1.out‚287622.1806640625# disk.dm-0.out‚287622.1806640625 disk.sda3.out‚ 696.15234375 disk.sda2.out‚0$ memory_percent‚93.15037727355957) proc.memory.percent‚17.94724464416504 disk.dm-0.in‚36546.04296875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 696.15234375 disk.sda2.in‚0( proc.memory.availableMB‚ 1122.2421875 proc.cpu.threads‚17`¨»’Ò,‚(:‡ ±éÔ„‡±´ gpu.0.gpu‚35.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚121.686' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OU„Ó,‚(:ù ±éÔæø¸( disk./.usagePercent‚78.8000227457574 disk.sda.out‚ 696.72265625 disk.sda2.out‚0( disk.nvme0c0n1.out‚287805.4970703125 disk.sda3.out‚ 696.72265625 disk.drbd1260.out‚ 1.7109375 proc.cpu.threads‚17% disk./.usageGB‚1174.1772842407227 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0n1.in‚36687.58984375& disk.nvme0n1.out‚287805.4970703125 disk.dm-0.in‚36687.58984375# disk.dm-0.out‚287805.4970703125$ disk.nvme0c0n1.in‚36687.58984375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 1122.7421875" proc.memory.rssMB‚ 2940.4765625) proc.memory.percent‚17.94724464416504 cpu‚25.315947032145303 network.sent‚ 2.9489961e+07 network.recv‚ 4.058784e+06 disk.sda1.in‚0 disk.dm-1.out‚ 696.72265625$ memory_percent‚93.14732551574707%vD4Ô,/ _wandbruntime‚12233  _runtime‚12233a¼\²ÉÕ,à & lr-AdamW/pg2‚6.764507165441186e-06 trainer/global_step‚11449 % _timestamp‚1.7885001335019865e+09 _runtime‚12233.142607143 _step‚939 & lr-AdamW/pg1‚6.764507165441186e-06îàê™ÎÖ,È _runtime‚12233.142607143 _step‚939 & lr-AdamW/pg1‚6.764507165441186e-06 & lr-AdamW/pg2‚6.764507165441186e-06 trainer/global_step‚11449 % _timestamp‚1.7885001335019865e+09«Ñd.Á4×,/ _wandbruntime‚12233  _runtime‚12233J[(Ø,‚P `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12272197157888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11771038633984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚812736821162496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚901665815123968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚800965800531456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11729434125136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63791979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11728663476724 % _timestamp‚1.7885001335027456e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚812737611810304 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11771038633984 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178792656 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚792189619507588 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11771811278848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚11449 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚812736821162496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚800965800531456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115000677 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 200571494 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 cÂà (Ù,‡P VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚812737611810304 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11771038633984 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178792656 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚792189619507588 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11771811278848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚11449 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚812736821162496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚800965800531456 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115000677 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 200571494 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12272197157888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11771038633984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚812736821162496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚901665815123968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚800965800531456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11729434125136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63791979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11728663476724 % _timestamp‚1.7885001335027456e+09¬ªÅÈ…4Ú,/ _wandbruntime‚12234  _runtime‚12234u½Â}Û,‹N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 254115843 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 200588370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178807023 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚780527022466624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 454681169 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚889468966308864 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11772816037888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 254115843 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚801033500470784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.788500134385667e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11772043379712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63796700 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚812805525847552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚11449 ô6D”TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63797226 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178807551 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚889468990248960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12273433278464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚792256687083226 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚780527039505984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11729664616602 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚801033482467840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 454704213 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚901742194972672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚941 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚812806316508672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚801033482467840 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 200588368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12273228663808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 454704213 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 454704213 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚901742194972672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚941 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚812806316508672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚801033482467840 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 200588368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12273228663808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 454704213 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178807023 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚780527022466624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 454681169 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚889468966308864 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11772816037888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 254115843 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚801033500470784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.788500134385667e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11772043379712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63796700 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚812805525847552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚11449 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63797226 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178807551 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚889468990248960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12273433278464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚792256687083226 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚780527039505984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11729664616602 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚801033482467840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115483075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚893161059045376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12324654567936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚804350797497856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 179540978 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚893161035105280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11822062979072 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚804350779494912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚905485897269760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11821290334208 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201415196 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 255183008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚816172069829120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12324838224384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _runtime‚12290.025167656 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 201415196 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚804350779494912 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11822062979072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64057901 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 255183008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183656448 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚795543799568468 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 trainer/global_step‚11499 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 456598204 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚905485689673216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F@DeviceStatsMonitor.on_traí¥ÒÐxin_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115483077 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚816172860476928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64057374 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11778734061832 % _timestamp‚1.7885001903844028e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚783764294858224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11821290334208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚816172069829120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115483075 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚893161059045376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12324654567936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚804350797497856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 179540978 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚893161035105280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11822062979072 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚804350779494912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚905485897269760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11821290334208 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201415196 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 255183008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚816172069829120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12324838224384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _runtime‚12290.025167656 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 201415196 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚804350779494912 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11822062979072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64057901 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 255183008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 183656448 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚795543799568468 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 trainer/global_step‚11499 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 456598204 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚905485689673216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 115483077°Éqxµ4î,/ _wandbruntime‚12291  _runtime‚12291<°È'ï,ŒN ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚804418479434240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 179555363 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _step‚945 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11779735201710 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 456636815 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚816241565175296 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11822295079936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11780505850330 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11822295079936 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚783830361293984 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11823067738112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚905562077293056 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 201432070 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 456613771 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11499 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201432068 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 255181703 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚804418497437184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 255204745 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚893236383448064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚795610079456334 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 456636815 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚783830344254624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664þ`ß'ð,‘N OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 201432068 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115492723 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚893236407388160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 201432070 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11823067738112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚816240774514176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12325693844992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚804418497437184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚816241565175296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 456636815 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚783830344254624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚804418479434240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 179555363 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 _step‚945 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11779735201710 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 456636815 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚816241565175296 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11822295079936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11780505850330 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11822295079936 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚783830361293984 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11823067738112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚905562077293056 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 201432070 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 456613771 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11499 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201432068 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 255181703 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚804418497437184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 255204745 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚893236383448064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚795610079456334 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11829575295352 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _step‚948 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 256248894 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚819607318495744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 202258894 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11828804646940 trainer/global_step‚11549 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 256271910 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115965475 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚798896404253804 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 256248894 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚807735776461312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚807735776461312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚798897191941576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 458530806 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚807735794464256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 202258894 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚10056zb (ÿ,‡P TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 458530806 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚807735794464256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 202258894 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚787067599606864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11871542034432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64324258 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 458507788 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 202258896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11829575295352 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 _step‚948 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 256248894 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚819607318495744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 202258894 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11828804646940 trainer/global_step‚11549 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 256271910 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115965475 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚798896404253804 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 256248894 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚807735776461312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚807735776461312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚798897191941576´8ãó¦4€-/ _wandbruntime‚12341  _runtime‚12341%ÿåŒ-ŒN I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 202275770 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 458546373 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 458569417 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 180305200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 202275768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚807803476400640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 256293647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11873319438336 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚807803494403584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚787133666042624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 256293647 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180304673 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚897003824527360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187837440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceSta;œí¯tsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64330075 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11829805786818 _runtime‚12341.371611713 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 458569417 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚819676813841920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚798964259517422 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 458546373 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211777536 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚819676023180800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115975123 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12378111709184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 _step‚949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚807803476400640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11872546780160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11872546780160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64329550 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚909382124073984 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992–Æ™ï'‚-‘N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 458546373 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211777536 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚819676023180800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115975123 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12378111709184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 _step‚949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚807803476400640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11872546780160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11872546780160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64329550 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚909382124073984 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 202275770 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 458546373 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 458569417 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 180305200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 202275768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚807803476400640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 256293647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11873319438336 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚807803494403584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚787133666042624 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 256293647 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180304673 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚897003824527360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187837440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64330075 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11829805786818 _runtime‚12341.371611713 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 458569417 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚819676813841920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚798964259517422 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0µÌñÁ#4ƒ-/ _wandbruntime‚12341  _runtime‚12341xØ a„-Û 0loss/SRqqt3/detection_loss‚1.668742045564087 %loss/total_loss‚39.40587916150159 $ _timestamp‚1.788500241733824e+09 0loss/FRt2/assignment_loss‚2.9380043251761077 0loss/FRt4/assignment_loss‚1.1042355388402938 _runtime‚12341.374189954 /loss/FRt4/detection_loss‚1.4072429513558746 _step‚950 .loss/FRt1/detection_loss‚2.453360802715733 2loss/SRqqt2/assignment_loss‚1.7706949085645054 2loss/SRqqt3/assignment_loss‚1.1387279963676344 0loss/FRt3/assignment_loss‚2.0039644489685693 2loss/SRqqt1/assignment_loss‚1.9510676691840803 2loss/SRqqt4/assignment_loss‚1.0681519831916109 .loss/FBt1/detection_loss‚2.422986014322801 /loss/FBt2/detection_loss‚2.5893618596924677 /loss/FBt4/detection_loss‚0.5951109950070862 2loss/FBt4/assignment_loss‚0.044405784779985163 /loss/FRt1/assignment_loss‚3.056411733229955 2loss/FBt1/assignment_loss‚0.013622814654307288 2loss/FBt3/assignment_loss‚0.014973841694768453 1loss/SRqqt1/detection_loss‚2.2272229258503233 1loss/SRqqt2/detection_loss‚2.5622225196465203 trainer/global_step‚11549 /loss/FRt2/detection_loss‚2.7878871392631295 1loss/FBt2/assignment_loss‚0.05425730483082387 .loss/FBt3/detection_loss‚2.261154394596815 epoch‚9 .loss/FRt3/detection_loss‚2.466259049044715 1loss/SRqqt4/detection_loss‚0.8058101149593911u‹¿{f…-à /loss/FRt2/detection_loss‚2.7878871392631295 1loss/FBt2/assignment_loss‚0.05425730483082387 .loss/FBt3/detection_loss‚2.261154394596815 epoch‚9 .loss/FRt3/detection_loss‚2.466259049044715 1loss/SRqqt4/detection_loss‚0.8058101149593911 0loss/SRqqt3/detection_loss‚1.668742045564087 %loss/total_loss‚39.40587916150159 $ _timestamp‚1.788500241733824e+09 0loss/FRt2/assignment_loss‚2.9380043251761077 0loss/FRt4/assignment_loss‚1.1042355388402938 _runtime‚12341.374189954 /loss/FRt4/detection_loss‚1.4072429513558746 _step‚950 .loss/FRt1/detection_loss‚2.453360802715733 2loss/SRqqt2/assignment_loss‚1.7706949085645054 2loss/SRqqt3/assignment_loss‚1.1387279963676344 0loss/FRt3/assignment_loss‚2.0039644489685693 2loss/SRqqt1/assignment_loss‚1.9510676691840803 2loss/SRqqt4/assignment_loss‚1.0681519831916109 .loss/FBt1/detection_loss‚2.422986014322801 /loss/FBt2/detection_loss‚2.5893618596924677 /loss/FBt4/detection_loss‚0.5951109950070862 2loss/FBt4/assignment_loss‚0.044405784779985163 /loss/FRt1/assignment_loss‚3.056411733229955 2loss/FBt1/assignment_loss‚0.013622814654307288 2loss/FBt3/assignment_loss‚0.014973841694768453 1loss/SRqqt1/detection_loss‚2.2272229258503233 1loss/SRqqt2/detection_loss‚2.5622225196465203 trainer/global_step‚11549¶1?¨5’†-‚(:‡ •²éÔ”¿ÿ° gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚161.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õœA€‡-‚(:õ •²éÔµµõ² disk.sda2.in‚0" disk.nvme0c0n1.in‚ 36809.359375 disk.nvme0n1.in‚ 36809.359375 disk.dm-1.in‚0 disk.dm-1.out‚ 700.3671875 proc.cpu.threads‚17 network.recv‚ 4.095523e+06% disk./.usageGB‚1174.4096412658691 disk.dm-0.in‚ 36809.359375 disk.drbd1260.out‚ 1.7265625 disk.sda.out‚ 700.3671875 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.94724464416504 network.sent‚ 2.9737694e+07 disk.sda2.out‚0' disk.nvme0n1.out‚290312.75634765625 disk.drbd1260.in‚0) proc.memory.availableMB‚ 1121.71484375 cpu‚25.314088640117347) disk.nvme0c0n1.out‚290312.75634765625$ disk.dm-0.out‚290312.75634765625 disk.sda3.in‚0 disk.sda3.out‚ 700.3671875 disk.sda.in‚0$ memory_percent‚93.15359592437744" proc.memory.rssMB‚ 2940.4765625) disk./.usagePercent‚78.81561642067551úŠ,çˆ-‚(:’ ¤²éÔ€ñ¡­ gpu.0.gpu‚51.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.384' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.876799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì ›ýƒ‰-‚(:ø ¤²éÔ„÷ã± disk.dm-1.out‚ 700.68359375 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0) proc.memory.percent‚17.94724464416504 disk.sda2.out‚0$ disk.nvme0c0n1.in‚36967.84765625& disk.nvme0n1.out‚290953.8330078125$ memory_percent‚93.15509796142578& proc.memory.availableMB‚ 1121.46875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.73046875 disk.sda.out‚ 700.68359375 disk.sda1.in‚0( disk.nvme0c0n1.out‚290953.8330078125 disk.dm-0.in‚36967.84765625 network.sent‚ 2.9848333e+07) disk./.usagePercent‚78.82557485774993" disk.nvme0n1.in‚36967.84765625# disk.dm-0.out‚290953.8330078125" proc.memory.rssMB‚ 2940.4765625 cpu‚25.31383559259921 proc.cpu.threads‚17 network.recv‚ 4.102085e+06% disk./.usageGB‚1174.5580291748047 disk.sda3.out‚ 700.68359375 l’Š-‚(:‡ ³²éÔøŠ½¯ gpu.0.gpu‚37.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.529' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.3058 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÝ ‡‹-‚(:ü ³²éÔþ­ï² disk.sda1.out‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚36967.84765625& disk.nvme0n1.out‚291136.8955078125 network.sent‚ 2.9853494e+07 disk.sda3.in‚0 disk.sda3.out‚ 701.05859375 disk.dm-1.out‚ 701.05859375 disk.sda.in‚0 disk.sda.out‚ 701.05859375 disk.sda2.in‚0 disk.dm-0.in‚36967.84765625) disk./.usagePercent‚78.83398214517437% disk./.usageGB‚1174.6833038330078 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.73046875( disk.nvme0c0n1.out‚291136.8955078125) proc.memory.availableMB‚ 1122.01953125" proc.memory.rssMB‚ 2940.4765625" disk.nvme0n1.in‚36967.84765625# disk.dm-0.out‚291136.8955078125$ memory_percent‚93.15173625946045) proc.memory.percent‚17.94724464416504 cpu‚25.313593268273642 proc.cpu.threads‚17 network.recv‚ 4.105619e+06 disk.sda1.in‚0ìŒ-‚(:’ ²éÔ𛡯 gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚123.175' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.634999999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N^Z„-‚(:ù ²éÔ‰‡ë²# disk.dm-0.out‚291393.4697265625 disk.dm-1.in‚0 disk.drbd1260.out‚1.734375 disk.sda1.out‚0) proc.memory.percent‚17.94724464416504 proc.cpu.threads‚17" disk.nvme0n1.in‚36967.84765625 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1122.2734375" proc.memory.rssMB‚ 2940.4765625 cpu‚25.313356663467022 network.sent‚ 2.9856715e+07) disk./.usagePercent‚78.85079364793287% disk./.usageGB‚1174.9338073730469& disk.nvme0n1.out‚291393.4697265625 disk.sda3.out‚ 701.58984375 disk.dm-1.out‚ 701.58984375 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚36967.84765625 network.recv‚ 4.109489e+06 disk.dm-0.in‚36967.84765625 disk.sda3.in‚0 disk.sda.out‚ 701.58984375 disk.sda1.in‚0( disk.nvme0c0n1.out‚291393.4697265625$ memory_percent‚93.15018653869629ÁìÀ×4Ž-/ _wandbruntime‚12391  _runtime‚12391,}-É-à % _timestamp‚1.7885002921438212e+09 _runtime‚12391.784384057 _step‚951 & lr-AdamW/pg1‚3.973704728970004e-06 & lr-AdamW/pg2‚3.973704728970004e-06 trainer/global_step‚11599̦ÕÎ-È & lr-AdamW/pg1‚3.973704728970004e-06 & lr-AdamW/pg2‚3.973704728970004e-06 trainer/global_step‚11599 % _timestamp‚1.7885002921438212e+09 _runtime‚12391.784384057 _step‚951·ݾ]>4‘-/ _wandbruntime‚12391  _runtime‚12391¼zì?÷&’-‚P H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚823043357810176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚559 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _step‚952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203102596 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211790848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚823042567162368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116447877 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12429800303616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚790370921394864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 116447875 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181039862 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚913125481836544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203102594 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811120791430656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚823042567162368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚11599 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11921793734656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚823043357810176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11921793734656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 257337796 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚802249796626912 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11879645880460 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚557 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚900695869383680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 181039303 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚802250584314684 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11922566379520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 460440390 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 257360812 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811120791430656 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125S\§” (“-‡P KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚823043357810176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚559 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 _step‚952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203102596 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211790848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚823042567162368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116447877 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12429800303616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚790370921394864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 116447875 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181039862 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚913125481836544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203102594 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811120791430656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚823042567162368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚11599 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11921793734656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚823043357810176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11921793734656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 257337796 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚802249796626912 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11879645880460 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚557 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚900695869383680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 181039303 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚802250584314684 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11922566379520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 460440390 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 257360812 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811120791430656 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516¸Ù\mˆ4”-/ _wandbruntime‚12392  _runtime‚12392üŸþ'•-‹N FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 460502019 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚802317651890530 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚900771217726464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11923571138560 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11922798480384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚913202103037952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚953 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 257359507 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 203119470 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12430673533952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚900771241666560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚802316864202550 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 460502019 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11922798480384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚12392.702165114 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203119468 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚913201891260416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚811188473367040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64597351 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187837440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 trainer/global_step‚11599 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181054333 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11880647020546 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12430861371392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64596810 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚811188491369984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 460478975 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 257382549 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11879876371926 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 460478975 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚823111271847424 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116457523 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 203119470 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 203119468 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 203119470 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 203119468 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 460502019 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚802317651890530 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚900771217726464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11923571138560 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11922798480384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚913202103037952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚953 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 257359507 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 203119470 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12430673533952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚900771241666560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚802316864202550 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 460502019 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11922798480384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚12392.702165114 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203119468 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚913201891260416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚811188473367040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64597351 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 187837440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 trainer/global_step‚11599 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181054333 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11880647020546 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12430861371392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825¹-N_ 4—-/ _wandbruntime‚12392  _runtime‚12392› k`˜-Ú 0loss/FRt1/assignment_loss‚3.2921641728816886 2loss/SRqqt1/assignment_loss‚1.0632800202885713 1loss/SRqqt1/detection_loss‚2.4816394492431924 3loss/FBt2/assignment_loss‚0.0010758555493642537 .loss/FRt3/detection_loss‚2.323308746166089 /loss/FBt3/detection_loss‚2.1289764881134032 /loss/FBt4/detection_loss‚0.5556385330009189 1loss/SRqqt4/detection_loss‚0.7757095374617275 2loss/SRqqt3/assignment_loss‚1.2272715151491689 2loss/SRqqt4/assignment_loss‚0.8964668402197555 /loss/FRt4/detection_loss‚1.4233376955682482 _runtime‚12392.703615998 0loss/FBt3/assignment_loss‚0.3151467760952009 1loss/SRqqt3/detection_loss‚1.8616850268913478 .loss/FBt1/detection_loss‚2.086648341268301 trainer/global_step‚11599 _step‚954 0loss/FRt3/assignment_loss‚1.9001145450507892 2loss/SRqqt2/assignment_loss‚1.9086837922704631 /loss/FRt2/detection_loss‚2.3448973754300293 epoch‚9 # _timestamp‚1.78850029306346e+09 0loss/FRt2/assignment_loss‚3.1814413051451407 /loss/FRt4/assignment_loss‚1.172121028105418 0loss/FBt1/assignment_loss‚0.5490730909514241 1loss/FBt4/assignment_loss‚0.25820912705768767 /loss/FRt1/detection_loss‚2.1119773235076513 0loss/SRqqt2/detection_loss‚2.130952291447541 &loss/total_loss‚39.218252068639394 /loss/FBt2/detection_loss‚3.2284331917762756¥û2»™-ß 2loss/SRqqt3/assignment_loss‚1.2272715151491689 2loss/SRqqt4/assignment_loss‚0.8964668402197555 /loss/FRt4/detection_loss‚1.4233376955682482 _runtime‚12392.703615998 0loss/FBt3/assignment_loss‚0.3151467760952009 1loss/SRqqt3/detection_loss‚1.8616850268913478 .loss/FBt1/detection_loss‚2.086648341268301 trainer/global_step‚11599 _step‚954 0loss/FRt3/assignment_loss‚1.9001145450507892 2loss/SRqqt2/assignment_loss‚1.9086837922704631 /loss/FRt2/detection_loss‚2.3448973754300293 epoch‚9 # _timestamp‚1.78850029306346e+09 0loss/FRt2/assignment_loss‚3.1814413051451407 /loss/FRt4/assignment_loss‚1.172121028105418 0loss/FBt1/aêA'(ªssignment_loss‚0.5490730909514241 1loss/FBt4/assignment_loss‚0.25820912705768767 /loss/FRt1/detection_loss‚2.1119773235076513 0loss/SRqqt2/detection_loss‚2.130952291447541 &loss/total_loss‚39.218252068639394 /loss/FBt2/detection_loss‚3.2284331917762756 0loss/FRt1/assignment_loss‚3.2921641728816886 2loss/SRqqt1/assignment_loss‚1.0632800202885713 1loss/SRqqt1/detection_loss‚2.4816394492431924 3loss/FBt2/assignment_loss‚0.0010758555493642537 .loss/FRt3/detection_loss‚2.323308746166089 /loss/FBt3/detection_loss‚2.1289764881134032 /loss/FBt4/detection_loss‚0.5556385330009189 1loss/SRqqt4/detection_loss‚0.7757095374617275º²Rœš-‚(:‘ ѲéÔÀÌê± gpu.0.gpu‚8.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.184' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.436799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉIìî„›-‚(:ù ѲéÔ…¡õ³$ memory_percent‚93.15059185028076) proc.memory.availableMB‚ 1122.20703125) proc.memory.percent‚17.94724464416504 cpu‚25.313095592294168 proc.cpu.threads‚17$ disk./.usageGB‚1174.932788848877 disk.sda3.in‚0 disk.drbd1260.out‚1.734375 disk.sda.out‚ 702.03515625 disk.sda1.out‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚36967.84765625" disk.nvme0n1.in‚36967.84765625 network.sent‚ 2.9862921e+07) disk./.usagePercent‚78.85072529392123 disk.sda3.out‚ 702.03515625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚291782.0322265625 network.recv‚ 4.113981e+06 disk.dm-1.out‚ 702.03515625 disk.sda.in‚0( disk.nvme0c0n1.out‚291782.0322265625 disk.dm-0.in‚36967.84765625" proc.memory.rssMB‚ 2940.4765625# disk.dm-0.out‚291782.0322265625¥Þœ-‚(:’ à²éÔ©—â® gpu.0.gpu‚38.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.149' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.629799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,Ü|-‚(:ñ à²éÔؽ¥³% disk./.usageGB‚1174.9093437194824 disk.sda1.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 1122.4609375) proc.memory.percent‚17.94724464416504 disk.sda1.in‚0 network.sent‚ 2.997153e+07 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 702.703125 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚292206.66259765625' disk.nvme0n1.out‚292206.66259765625 network.recv‚ 4.120119e+06 disk.drbd1260.out‚ 1.7421875 disk.sda.out‚ 702.703125! disk.nvme0n1.in‚ 37097.7734375$ disk.dm-0.out‚292206.66259765625# memory_percent‚93.1490421295166 cpu‚25.31286255286494 proc.cpu.threads‚17) disk./.usagePercent‚78.84915187161579 disk.sda3.out‚ 702.703125 disk.sda.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 37097.7734375 disk.dm-0.in‚ 37097.7734375" proc.memory.rssMB‚ 2940.4765625HïvN’ž-‚(:‡ ï²éÔóË£® gpu.0.gpu‚16.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.471' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©±¤~Ÿ-‚(:ó ï²éÔ²ž² disk.sda3.in‚0 disk.drbd1260.out‚ 1.7421875 disk.sda.out‚ 703.078125 disk.sda2.in‚0) disk.nvme0c0n1.out‚292655.18994140625' disk.nvme0n1.out‚292655.18994140625$ disk.dm-0.out‚292655.18994140625 disk.sda1.out‚0 network.sent‚ 2.9976752e+07 network.recv‚ 4.123667e+06% disk./.usageGB‚1175.4348106384277 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 703.078125! disk.nvme0n1.in‚ 37097.7734375 disk.dm-0.in‚ 37097.7734375$ memory_percent‚93.15061569213867' proc.memory.availableMB‚ 1122.203125) proc.memory.percent‚17.94724464416504 cpu‚25.312631897105067 proc.cpu.threads‚17 disk.sda1.in‚0) disk./.usagePercent‚78.88441639744231 disk.sda3.out‚ 703.078125# disk.nvme0c0n1.in‚ 37097.7734375" proc.memory.rssMB‚ 2940.4765625´'ô4 -/ _wandbruntime‚12443  _runtime‚12443ëÇä É¡-à % _timestamp‚1.7885003439954913e+09 _runtime‚12443.636026953 _step‚955 & lr-AdamW/pg1‚3.206641577495694e-06 & lr-AdamW/pg2‚3.206641577495694e-06 trainer/global_step‚11649$Hѳ΢-È & lr-AdamW/pg1‚3.206641577495694e-06 & lr-AdamW/pg2‚3.206641577495694e-06 trainer/global_step‚11649 % _timestamp‚1.7885003439954913e+09 _runtime‚12443.636026953 _step‚955»@q4£-/ _wandbruntime‚12443  _runtime‚12443&XR/(¤-‚P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 181790543 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚904463310462976 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚805603976687792 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚826477815828992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚814505788397056 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 _step‚956 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 203946296 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 258426698 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 258426698 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11972818079744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚814505770394112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 462396010 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 258449714 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181791100 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚916945661004288 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 462372992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚814505788397056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12482350541312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚826478606476800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11929716465568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 116930275 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11972045434880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚826477815828992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚814505788397056 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 _step‚956 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 203946296 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 258426698 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 258426698 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11972818079744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚814505770394112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 462396010 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 258449714 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181791100 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚916945661004288 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 462372992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚814505788397056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12482350541312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚826478606476800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11929716465568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 116930275 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11972045434880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 B±Ò4¦-/ _wandbruntime‚12444  _runtime‚12444m`ß’\§-ŒN RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 203963170 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 462411577 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203963168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 462434621 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚826547311175168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462434621 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 258471451 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 258448409 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚826546520514048 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11973822838784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194128896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12483210979840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11973050180608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 258448409 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11929946957034 % _timestamp‚1.7885003449242325e+09 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116939923 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 258471451 _runtime‚12444.565191271 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 181805976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚904538658805760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64866051 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚11649 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 203963168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚826547311175168 VHDeviceStatsMonitor.on_trainÔU¤¶ _batch_end/allocated_bytes.small_pool.current‚ 772658176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚805670256575658 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181805431 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64865508 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11973822838784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12483405108736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚917022063914496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352}VŠ'¨-‘N ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 462434621 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚826547311175168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462434621 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 258471451 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 258448409 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚826546520514048 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11973822838784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 218068992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 194128896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12483210979840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11973050180608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 258448409 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11929946957034 % _timestamp‚1.7885003449242325e+09 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116939923 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 258471451 _runtime‚12444.565191271 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 181805976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚904538658805760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64866051 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚11649 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 203963168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚826547311175168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚805670256575658 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181805431 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64865508 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11973822838784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12483405108736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚917022063914496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 203963170 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 462411577 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203963168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 259538616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚920765247427584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 464328612 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117412677 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117412675 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65127865 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚908230727602176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚817890785363456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12534735810560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚808956581373128 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11979787050676 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 259538616 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 259515600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚829913064495616 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12022297135104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚829913064495616 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚920765463412736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 259515600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12023069779968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚12493.041001026 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚796977582010224 _step‚960 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772"?烷-‡P O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 464305594 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚829913855143424 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12022297135104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12534543765504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚908230703662080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 204789996 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚817890767360512 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 182540542 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885003934003053e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚817890785363456 XJDeviceStatsMonitor.on_train_batch_start/allocated_byte²˜Šs.small_pool.current‚ 772644864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12023069779968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚829913855143424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204789996 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚796977564970864 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 204789994 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11979016402264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 259538616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚920765247427584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 464328612 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117412677 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117412675 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65127865 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚908230727602176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚817890785363456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12534735810560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚808956581373128 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11979787050676 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 259538616 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 259515600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚829913064495616 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12022297135104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚829913064495616 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚920765463412736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 259515600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12023069779968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚12493.041001026 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚796977582010224 _step‚960 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 464367223 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11980788190762 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12023301880832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12535787392000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259560353 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 464344179 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204806870 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 204806870 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚817958467299840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12024074539008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65133091 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚908306052004864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚829981769180672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚920841863336960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 464367223 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11699 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚817958485302784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182554842 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚12494.135367851 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 464344179 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 259537311 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚829982559841792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚809024436636746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11980017542142 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12024074539008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚961 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204806868 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 259537311 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚829982559841792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚809024436636746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11980017542142 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12024074539008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚961 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204806868 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 464367223 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11980788190762 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12023301880832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12535787392000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259560353 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 464344179 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204806870 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 204806870 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚817958467299840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12024074539008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65133091 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚908306052004864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚829981769180672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚920841863336960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 464367223 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11699 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚817958485302784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182554842 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 _runtime‚12494.135367851 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 464344179 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚911998120801280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117895077 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚821275782329856 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12072548835328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 205633694 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 183290444 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚833348313162240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚800280903798224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚812309973746236 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚924585044524544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 _runtime‚12543.623483398 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚800280886758864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 260627518 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12073321480192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚911998144741376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12073321480192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 trainer/global_step‚11749 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 205633694 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65395367 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 466261214 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚833348313162240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65394812 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 205633696 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183289887 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚924585264704000 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12586923723264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005°šÙ (Ë-‡P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚821275782329856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚821275764326912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12587119962624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 260604502 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚833348313162240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚800280903798224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚812309973746236 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚924585044524544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 196239360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 _runtime‚12543.623483398 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚800280886758864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 260627518 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12073321480192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚911998144741376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12073321480192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 trainer/global_step‚11749 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 205633694 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65395367 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 466261214 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚833348313162240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65394812 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 205633696 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183289887 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚924585264704000 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12586923723264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 205633696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚812310761434008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12029086987372 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 117895075 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚911998120801280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117895077 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚821275782329856 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12072548835328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 205633694 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 183290444 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12074326239232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚800346953194624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 466299825 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183304977 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚924661659662336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚821343482269184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 260649255 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚800346970233984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 % _timestamp‚1.7885004452307577e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 466299825 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 466276781 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117904725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65400254 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚912073493084160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205650568 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12073553581056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 260626213 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚833417808508416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 466276781 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚555 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117904723 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12030858775870 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ?6DeviceStat •Eû(sMonitor.on_train_batch_end/segment.all.peak‚1005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚12544.872239325 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚833417808508416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65400807 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 205650570 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312¿'Î-‘N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 466276781 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117904725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65400254 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204614656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚912073493084160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205650568 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12073553581056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 260626213 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚833417808508416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 466276781 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚555 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117904723 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12030858775870 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚12544.872239325 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚833417808508416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65400807 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 205650570 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12074326239232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 260649255 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 trainer/global_step‚11749 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚924661431107584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚821343464266240 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183305532 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚812377829009854 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 205650568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚912073469144064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12587961963520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚821343464266240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚833417017847296 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚821343482269184 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12073553581056 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228554752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚965 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12074326239232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚800346953194624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 466299825 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183304977 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚924661659662336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚821343482269184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 260649255 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚800346970233984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 % _timestamp‚1.7885004452307577e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 466299825 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312ÅQÊD4Ï-/ _wandbruntime‚12544  _runtime‚12544A2¿»\Ð-Ö 0loss/FBt1/assignment_loss‚0.2627230687843009 /loss/FBt3/detection_loss‚1.7874726891517638 0loss/FRt4/assignment_loss‚1.0272967581655466 /loss/FRt1/detection_loss‚2.2068310448692903 0loss/SRqqt1/detection_loss‚2.167091619027288 .loss/FBt2/detection_loss‚2.659605880578359 $loss/total_loss‚37.9166450677692 $ _timestamp‚1.788500445234458e+09 2loss/SRqqt3/assignment_loss‚1.3186745537444948 .loss/FRt2/detection_loss‚2.559377404608844 /loss/FRt3/detection_loss‚2.5753623413688995 1loss/SRqqt2/detection_loss‚2.0373072833628267 trainer/global_step‚11749 2loss/SRqqt1/assignment_loss‚1.2830012883304764 0loss/FBt3/assignment_loss‚0.2270394153343652 /loss/FRt4/detection_loss‚1.6788091969073695 1loss/SRqqt3/detection_loss‚1.8399797601625323 _runtime‚12544.874798248 2loss/SRqqt2/assignment_loss‚1.4834544183434666 epoch‚9 0loss/FRt3/assignment_loss‚3.0323973052641926 1loss/SRqqt4/assignment_loss‚0.692162075622336 /loss/FBt1/detection_loss‚1.9847245713075001 _step‚966 0loss/FRt1/assignment_loss‚2.6515794144736398 /loss/FRt2/assignment_loss‚2.723246388965183 0loss/FBt2/assignment_loss‚0.2356435854665809 2loss/FBt4/assignment_loss‚0.059060994958978576 1loss/SRqqt4/detection_loss‚0.9101845484007808 /loss/FBt4/detection_loss‚0.5136194605701825y*daÑ-Û 0loss/FBt2/assignment_loss‚0.2356435854665809 2loss/FBt4/assignment_loss‚0.059060994958978576 1loss/SRqqt4/detection_loss‚0.9101845484007808 /loss/FBt4/detection_loss‚0.5136194605701825 0loss/FBt1/assignment_loss‚0.2627230687843009 /loss/FBt3/detection_loss‚1.7874726891517638 0loss/FRt4/assignment_loss‚1.0272967581655466 /loss/FRt1/detection_loss‚2.2068310448692903 0loss/SRqqt1/detection_loss‚2.167091619027288 .loss/FBt2/detection_loss‚2.659605880578359 $loss/total_loss‚37.9166450677692 $ _timestamp‚1.788500445234458e+09 2loss/SRqqt3/assignment_loss‚1.3186745537444948 .loss/FRt2/detection_loss‚2.559377404608844 /loss/FRt3/detection_loss‚2.5753623413688995 1loss/SRqqt2/detection_loss‚2.0373072833628267 trainer/global_step‚11749 2loss/SRqqt1/assignment_loss‚1.2830012883304764 0loss/FBt3/assignment_loss‚0.2270394153343652 /loss/FRt4/detection_loss‚1.6788091969073695 1loss/SRqqt3/detection_loss‚1.8399797601625323 _runtime‚12544.874798248 2loss/SRqqt2/assignment_loss‚1.4834544183434666 epoch‚9 0loss/FRt3/assignment_loss‚3.0323973052641926 1loss/SRqqt4/assignment_loss‚0.692162075622336 /loss/FBt1/detection_loss‚1.9847245713075001 _step‚966 0loss/FRt1/assignment_loss‚2.6515794144736398 /loss/FRt2/assignment_loss‚2.723246388965183ÆÉvØŠ’Ò-‚(:‡ ç³éÔæüß­ gpu.0.gpu‚48.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚173.209' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,_‚Ó-‚(:÷ ç³éÔ ÷ì± disk.sda3.in‚0 disk.sda3.out‚ 706.796875 disk.drbd1260.out‚ 1.7578125 disk.sda.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚295639.13330078125' disk.nvme0n1.out‚295639.13330078125 network.recv‚ 4.160768e+06 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚37376.10546875$ memory_percent‚93.15288066864014) proc.memory.percent‚17.94724464416504 cpu‚25.31069313345106 proc.cpu.threads‚17) disk./.usagePercent‚78.80825978817158 disk.dm-1.out‚ 706.796875 disk.drbd1260.in‚0 disk.sda.out‚ 706.796875" proc.memory.rssMB‚ 2940.4765625 network.sent‚ 3.0227461e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.in‚37376.10546875 disk.dm-0.in‚37376.10546875$ disk.dm-0.out‚295639.13330078125) proc.memory.availableMB‚ 1121.83203125% disk./.usageGB‚1174.3000221252441`qZ’Ô-‚(:‡ ö³éÔ¿„ò¬ gpu.0.gpu‚35.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.471' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.4942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 úFÓ‚Õ-‚(:÷ ö³éÔß‘è¯) disk./.usagePercent‚78.80082763344505 disk.sda2.out‚0$ disk.nvme0c0n1.in‚37376.10546875) disk.nvme0c0n1.out‚296001.25244140625 disk.sda.out‚ 707.1640625$ memory_percent‚93.15471649169922 proc.cpu.threads‚17 network.recv‚ 4.165464e+06 disk.sda1.out‚0" disk.nvme0n1.in‚37376.10546875' disk.nvme0n1.out‚296001.25244140625 disk.sda3.out‚ 707.1640625 disk.dm-1.out‚ 707.1640625 disk.drbd1260.out‚1.765625% disk./.usageGB‚1174.1892776489258 disk.dm-0.in‚37376.10546875$ disk.dm-0.out‚296001.25244140625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0" proc.memory.rssMB‚ 2940.4765625) proc.memory.percent‚17.94724464416504 network.sent‚ 3.0334239e+07 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.in‚0& proc.memory.availableMB‚ 1121.53125 cpu‚25.310451737848997éBŠ€Ö-‚(:’ …´éÔ—Ÿä® gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.781' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.156200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±G%„×-‚(:ù …´éÔ¾ñé°$ disk.nvme0c0n1.in‚37376.10546875) disk.nvme0c0n1.out‚296475.85791015625 disk.dm-0.in‚37376.10546875 disk.sda3.out‚ 707.59765625 disk.dm-1.in‚0 disk.sda.out‚ 707.59765625) proc.memory.percent‚17.94724464416504 cpu‚25.31019826713234 network.sent‚ 3.0340547e+07 network.recv‚ 4.170246e+06) disk./.usagePercent‚78.81104899025712% disk./.usageGB‚1174.3415832519531 disk.sda1.in‚0" disk.nvme0n1.in‚37376.10546875 disk.dm-1.out‚ 707.59765625 disk.drbd1260.out‚1.765625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚296475.85791015625$ disk.dm-0.out‚296475.85791015625 disk.sda.in‚0$ memory_percent‚93.15319061279297& proc.memory.availableMB‚ 1121.78125 disk.sda3.in‚0 disk.drbd1260.in‚0" proc.memory.rssMB‚ 2940.4765625 proc.cpu.threads‚17LÔ5Z4Ø-/ _wandbruntime‚12596  _runtime‚12596âJ•GËÙ-Å _step‚967 ' lr-AdamW/pg1‚1.3973028392882434e-06 ' lr-AdamW/pg2‚1.3973028392882434e-06 trainer/global_step‚11799 % _timestamp‚1.7885004964362476e+09 _runtime‚12596.076711043ý̲hÐÚ-Ê ' lr-AdamW/pg1‚1.3973028392882434e-06 ' lr-AdamW/pg2‚1.3973028392882434e-06 trainer/global_step‚11799 % _timestamp‚1.7885004964362476e+09 _runtime‚12596.076711043 _step‚967ǸÆß4Û-/ _wandbruntime‚12596  _runtime‚12596P¦˜(Ü-‚P F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚968 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 206477396 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 206477394 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12639364513280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65662780 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12079157572480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚836784352476672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12123573180416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚803584225586224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚824660779296256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 206477396 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12122800535552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12079928220892 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 261716420 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚803584208546864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12123573180416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 261693404 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚836783561828864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12639560752640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚824660761293312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚836784352476672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 206477394 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118377475 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118377477 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 468170798 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚836783561828864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12122800535552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚915765561880576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚824660761293312 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 184039703 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65662228 _runtime‚12596.078159653 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 261716420 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚928405122633216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚915765537940480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚824660779296256 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 184040257 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚928404902453760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118377477 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 468170798 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚836783561828864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12122800535552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚915765561880576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚824660761293312 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 184039703 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65662228 _runtime‚12596.078159653 QCDeviceStatsMonitor.on_train_batch_sta[×hÜrt/active.small_pool.allocated‚ 261716420 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚928405122633216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚915765537940480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚824660779296256 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 184040257 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220179456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚928404902453760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚803584208546864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12123573180416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 261693404 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚836783561828864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624Èå÷ ‹4Þ-/ _wandbruntime‚12596  _runtime‚12596CÝmÍ'ß-ŒN D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚824728479235584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12124577939456 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 206494268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 % _timestamp‚1.7885004973485239e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184054633 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚824728461232640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚824728461232640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 468209383 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 184055184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65667510 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚11799 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 468232427 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 468232427 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚803650274982624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 261715115 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚551 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118387125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65668059 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 261715115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚549 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12640406814208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚836853057175040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚836852266513920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚928481517457920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 206494270 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12123805281280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚803650292021984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 261738157 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚815731221382962 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 206494268 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 468209383 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚915840886283264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚12596.989520159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚824728479235584 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118387123]*—Ì'à-‘N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 468232427 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚803650274982624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 261715115 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚551 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118387125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65668059 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 261715115 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚549 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12640406814208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚836853057175040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚836852266513920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚928481517457920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 206494270 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12123805281280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚803650292021984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 261738157 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚815731221382962 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224360448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200420352 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 206494268 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 468209383 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚915840886283264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚12596.989520159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚824728479235584 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118387123 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚824728479235584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12124577939456 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 206494268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 % _timestamp‚1.7885004973485239e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184054633 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚824728461232640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚824728461232640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 468209383 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 184055184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65667510 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚11799 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 468232427 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840218810495488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚840218810495488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚828045776262656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12692106948608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚12645.172772078 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚828045758259712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 % _timestamp‚1.7885005455320454e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12173824880640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 207321094 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚819016758492452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118859877 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12173052235776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 207321096 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 262782306 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12173052235776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚828045758259712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12173824880640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65927428 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118859875 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚806887547374224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚840219601143296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 470103400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 262805322 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65927957 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚819017546180224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12129998806000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚972 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚919532955079680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 262782306 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12129228157588 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 trainer/global_step‚11849óGxô (ñ-‡P QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118859875 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚806887547374224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚840219601143296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 470103400 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 262805322 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65927957 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚819017546180224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12129998806000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚972 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚919532955079680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 262782306 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12129228157588 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 trainer/global_step‚11849 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚840219601143296 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207321096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207321094 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚932225085968384 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840218810495488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚840218810495488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚828045776262656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224373760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12692106948608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚12645.172772078 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚828045758259712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 % _timestamp‚1.7885005455320454e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12173824880640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 207321094 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200433664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚819016758492452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118859877 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12173052235776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 207321096 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 262782306 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12173052235776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚828045758259712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12173824880640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65927428ÌéTê4ò-/ _wandbruntime‚12646  _runtime‚12646œñ~o$ó-ŒN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 207337968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚840288305841664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12174056981504 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 207337970 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 _step‚973 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚828113458199040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚828113476201984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65932688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚828113458199040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 470141985 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚840287515180544 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 184802746 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184802211 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚806953613809984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118869523 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 470165029 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚828113476201984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚12646.277293231 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚806953596770624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚919608327362560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 trainer/global_step‚11849 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12174056981504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12692966044672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 207337970 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 262804017 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 470165029 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 207337968 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118869525 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚919608303422464 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12174829639680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 470165029 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚828113476201984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚12646.277293231 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚806953596770624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚919608327362560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 trainer/global_step‚11849 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12174056981504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12692966044672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 207337970 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 262804017 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 470165029 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 207337968 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118869525 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚919608303422464 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12174829639680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚840288305841664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12174056981504 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 207337970 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 _step‚973 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚828113458199040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚828113476201984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65932688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚828113458199040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 470141985 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚840287515180544 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 184802746 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184802211 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688ÍÓMñ4õ-/ _wandbruntime‚12646  _runtime‚12646ñJTbö-Ü /loss/FRt1/assignment_loss‚2.705935465438025 2loss/SRqqt2/assignment_loss‚1.6293705376711758 1loss/FBt3/assignment_loss‚0.00065778195725324 /loss/FRt4/detection_loss‚1.4418774566760189 1loss/SRqqt4/detection_loss‚0.6814388588032464 _runtime‚12646.280428399 2loss/FBt2/assignment_loss‚0.040473714277335864 1loss/SRqqt3/detection_loss‚1.9409622194414788 .loss/FBt3/detection_loss‚2.672041876749559 % _timestamp‚1.7885005466398294e+09 2loss/SRqqt1/assignment_loss‚1.5597420970776252 2loss/SRqqt3/assignment_loss‚1.3251609064807945 1loss/FBt1/assignment_loss‚0.11825576847275862 0loss/SRqqt1/detection_loss‚2.545028697167124 0loss/FRt3/assignment_loss‚2.1125002485864304 1loss/FBt4/assignment_loss‚0.09955450536028292 /loss/FBt1/detection_loss‚2.1095905487353983 /loss/FBt4/detection_loss‚0.4829324218282086 epoch‚9 /loss/FRt1/detection_loss‚2.2604516223073006 trainer/global_step‚11849 0loss/FRt4/assignment_loss‚1.4024993536765116 1loss/SRqqt2/detection_loss‚2.0800259176528817 _step‚974 0loss/FRt2/assignment_loss‚2.5429683719810687 /loss/FRt2/detection_loss‚2.5062258648329974 %loss/total_loss‚38.82288814155553 2loss/SRqqt4/assignment_loss‚1.2376529860703909 /loss/FRt3/detection_loss‚2.5374575800755443 .loss/FBt2/detection_loss‚2.790083340236119ÚÎYOg÷-á 0loss/FRt3/assignment_loss‚2.1125002485864304 1loss/FBt4/assignment_loss‚0.09955450536028292 /loss/FBt1/detection_loss‚2.1095905487353983 /loss/FBt4/detection_loss‚0.4829324218282086 epoch‚9 /loss/FRt1/detection_loss‚2.2604516223073006 trainer/global_step‚11849 0loss/FRt4/assignment_loss‚1.4024993536765116 1loss/SRqqt2/detection_loss‚2.0800259176528817 _step‚974 0loss/FRt2/assignment_loss‚2.5429683719810687 /loss/FRt2/detection_loss‚2.5062258648329974 %loss/total_loss‚38.82288814155553 2loss/SRqqt4/assignment_loss‚1.2376529860703909 /loss/FRt3/detection_loss‚2.5374575800755443 .loss/FBt2/detection_loss‚2.790083340236119 /loss/FRt1/assignment_loss‚2.705935465438025 2loss/SRqqt2/assignment_loss‚1.6293705376711758 1loss/FBt3/assignment_loss‚0.00065778195725324 /loss/FRt4/detection_loss‚1.4418774566760189 1loss/SRqqt4/detection_loss‚0.6814388588032464 _runtime‚12646.280428399 2loss/FBt2/assignment_loss‚0.040473714277335864 1loss/SRqqt3/detection_loss‚1.9409622194414788 .loss/FBt3/detection_loss‚2.672041876749559 % _timestamp‚1.7885005466398294e+09 2loss/SRqqt1/assignment_loss‚1.5597420970776252 2loss/SRqqt3/assignment_loss‚1.3251609064807945 1loss/FBt1/assignment_loss‚0.11825576847275862 0loss/SRqqt1/detection_loss‚2.545028697167124ÎŽÌZ’ø-‚(:‡ дéԽ gpu.0.gpu‚31.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚127.607' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôQ-yù-‚(:î дéÔº˜³ disk.sda.out‚ 710.1171875 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚93.15376281738281 proc.cpu.threads‚17$ disk./.usageGB‚1174.437385559082& disk.nvme0c0n1.out‚298203.43359375" disk.nvme0n1.in‚37616.55078125 disk.dm-0.in‚37616.55078125 disk.drbd1260.out‚ 1.77734375 disk.sda.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.94724464416504 network.sent‚ 3.0472048e+07 disk.sda3.out‚ 710.1171875 disk.sda2.out‚0" proc.memory.rssMB‚ 2940.4765625 cpu‚25.308971522736098 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 710.1171875 network.recv‚ 4.193803e+06) disk./.usagePercent‚78.81747836347203$ disk.nvme0n1.out‚298203.43359375! disk.dm-0.out‚298203.43359375% proc.memory.availableMB‚ 1121.6875$ disk.nvme0c0n1.in‚37616.55078125 disk.drbd1260.in‚0))#–’ú-‚(:‡ ß´éÔã‹‹¯ gpu.0.gpu‚46.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.9386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gDÿvû-‚(:ë ß´éÔØæ ² network.sent‚ 3.0580024e+07 network.recv‚ 4.199711e+06% disk./.usageGB‚1174.4745559692383 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚298627.8671875 cpu‚25.30874849539613 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0n1.out‚298627.8671875 disk.dm-0.out‚298627.8671875$ memory_percent‚93.15223693847656% proc.memory.availableMB‚ 1121.9375) proc.memory.percent‚17.94724464416504 proc.cpu.threads‚17) disk./.usagePercent‚78.81997290088569 disk.sda3.out‚ 710.6171875 disk.sda2.out‚0" disk.nvme0n1.in‚37661.55078125 disk.dm-0.in‚37661.55078125" proc.memory.rssMB‚ 2940.4765625 disk.dm-1.in‚0 disk.dm-1.out‚ 710.6171875 disk.drbd1260.out‚ 1.77734375 disk.sda.in‚0 disk.sda.out‚ 710.6171875$ disk.nvme0c0n1.in‚37661.55078125†¹ P’ü-‚(:‡ î´éÔÆç® gpu.0.gpu‚38.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.423' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8846 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ4–û}ý-‚(:ò î´éÔ–ð² network.recv‚ 4.20339e+06( disk./.usagePercent‚78.8299367141183% disk./.usageGB‚1174.6230239868164 disk.dm-1.out‚ 711.0234375 disk.sda.in‚0) disk.nvme0c0n1.out‚299076.66748046875 disk.dm-0.in‚37661.55078125$ disk.dm-0.out‚299076.66748046875 disk.drbd1260.out‚1.78125 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2940.4765625) proc.memory.percent‚17.94724464416504 network.sent‚ 3.0585202e+07 disk.sda3.out‚ 711.0234375 disk.drbd1260.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚37661.55078125" disk.nvme0n1.in‚37661.55078125$ memory_percent‚93.15223693847656 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 711.0234375 disk.sda2.in‚0' disk.nvme0n1.out‚299076.66748046875% proc.memory.availableMB‚ 1121.9375 cpu‚25.30852241660445wskÔ4þ-/ _wandbruntime‚12696  _runtime‚12696 ÏšzÈÿ- & lr-AdamW/pg1‚6.021666876357889e-07 & lr-AdamW/pg2‚6.021666876357889e-07 trainer/global_step‚11899 $ _timestamp‚1.788500597072155e+09 _runtime‚12696.712445868 _step‚975~“Í€.Ç & lr-AdamW/pg1‚6.021666876357889e-07 & lr-AdamW/pg2‚6.021666876357889e-07 trainer/global_step‚11899 $ _timestamp‚1.788500597072155e+09 _runtime‚12696.712445868 _step‚975Ï# ì4./ _wandbruntime‚12696  _runtime‚12696;²wÖ(‚.P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚936044832102400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12744651928576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚831430755226112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 $ _timestamp‚1.788500597073156e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚831430755226112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚843654849809920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚843654059162112 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12224076580864 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12180069391108 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 185536298 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚843654059162112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚822370150865560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 208164794 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12224076580864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12744459883520 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66194021 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 263871208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚810190852122864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚831430773229056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185535752 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 472036002 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 263894224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12223303936000©á^¬ƒ.†P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚831430755226112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 $ _timestamp‚1.788500597073156e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚831430755226112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚843654849809920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚843654059162112 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12224076580864 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12180069391108 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 185536298 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚843654059162112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚822370150865560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 208164794 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12224076580864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12744459883520 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66194021 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1025507328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 263871208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215985152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚810190852122864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚831430773229056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185535752 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 472036002 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 263894224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12223303†9èK`936000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚936044832102400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12744651928576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192045056 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412Ðv‹ª4„./ _wandbruntime‚12697  _runtime‚12697íkè'….ŒN \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚936121451813888 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12224308681728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 263892919 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12181070531194 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 472097631 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66199396 trainer/global_step‚11899 % _timestamp‚1.7885005979972148e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 263915961 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12224308681728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 208181670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213874688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚831498455165440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 263892919 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12745707312128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12225081339904 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185550764 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚822438006129178 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208181670 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208181668 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚923375744501760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 _step‚977 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 472074587 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚831498473168384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12180299882574 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119351925 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12225081339904 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185550764 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189934592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚822438006129178 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208181670 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208181668 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚923375744501760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 _step‚977 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 472074587 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚831498473168384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12180299882574 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119351925 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 263892919 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12181070531194 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 472097631Ñ‚™$/4‡./ _wandbruntime‚12697  _runtime‚126975|ø¹]ˆ.× /loss/FBt3/detection_loss‚2.4093775550524392 1loss/SRqqt3/assignment_loss‚1.253945354534232 1loss/FBt1/assignment_loss‚0.10290808215114487 epoch‚9 trainer/global_step‚11899 /loss/FRt2/detection_loss‚2.4065351129909938 1loss/SRqqt1/assignment_loss‚1.834853912031071 1loss/SRqqt2/assignment_loss‚1.376549457234365 1loss/FBt4/assignment_loss‚0.10964606127031255 /loss/FBt4/detection_loss‚0.6701121195144351 2loss/FBt3/assignment_loss‚0.020282771790481508 $ _timestamp‚1.788500597999583e+09 /loss/FRt1/assignment_loss‚2.738932251930237 2loss/SRqqt4/assignment_loss‚0.8102245429074582 1loss/SRqqt1/detection_loss‚2.1749219596385956 /loss/FBt2/detection_loss‚2.1831813355286918 _step‚978 .loss/FRt3/detection_loss‚2.371628089911408 1loss/SRqqt3/detection_loss‚1.7941496265323267 /loss/FBt1/detection_loss‚2.2278787162568836 &loss/total_loss‚38.183830568122204 0loss/FRt3/assignment_loss‚2.7039426730738745 0loss/FRt4/assignment_loss‚0.9244340981046358 /loss/FRt4/detection_loss‚1.2880073956524332 _runtime‚12697.639780103 0loss/FBt2/assignment_loss‚0.1939694883051541 1loss/SRqqt4/detection_loss‚0.8794330166348693 .loss/FRt1/detection_loss‚2.402057070539079 /loss/FRt2/assignment_loss‚3.034155171493004 0loss/SRqqt2/detection_loss‚2.272704705044075't€b‰.Ü 2loss/SRqqt4/assignment_loss‚0.8102245429074582 1loss/SRqqt1/detection_loss‚2.1749219596385956 /loss/FBt2/detection_loss‚2.1831813355286918 _step‚978 .loss/FRt3/detection_loss‚2.371628089911408 1loss/SRqqt3/detection_loss‚1.7941496265323267 /loss/FBt1/detection_loss‚2.2278787162568836 &loss/total_loss‚38.183830568122204 0loss/FRt3/assignment_loss‚2.7039426730738745 0loss/FRt4/assignment_loss‚0.9244340981046358 /loss/FRt4/detection_loss‚1.2880073956524332 _runtime‚12697.639780103 0loss/FBt2/assignment_loss‚0.1939694883051541 1loss/SRqqt4/detection_loss‚0.8794330166348693 .loss/FRt1/detection_loss‚2.402057070539079 /loss/FRt2/assignment_loss‚3.034155171493004 0loss/SRqqt2/detection_loss‚2.272704705044075 /loss/FBt3/detection_loss‚2.4093775550524392 1loss/SRqqt3/assignment_loss‚1.253945354534232 1loss/FBt1/assignment_loss‚0.10290808215114487 epoch‚9 trainer/global_step‚11899 /loss/FRt2/detection_loss‚2.4065351129909938 1loss/SRqqt1/assignment_loss‚1.834853912031071 1loss/SRqqt2/assignment_loss‚1.376549457234365 1loss/FBt4/assignment_loss‚0.10964606127031255 /loss/FBt4/detection_loss‚0.6701121195144351 2loss/FBt3/assignment_loss‚0.020282771790481508 $ _timestamp‚1.788500597999583e+09 /loss/FRt1/assignment_loss‚2.738932251930237ÒÑDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 209008494 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 186284473 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚825723543238668 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 % _timestamp‚1.7885006457085848e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚847089307828736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚847090098476544 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 186285029 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230665216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚813494190950224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚980 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119824675 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12273555636224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206725120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚813494173910864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 473968604 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 264960110 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12229369327804 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚939864961719808 trainer/global_step‚11949 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 209008496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 264960110 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12796941696512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚554 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264983126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12274328281088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚927067789358080 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66460352˜Q“Š (•.‡P aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚927067789358080 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66460352 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 209008494 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚834815770195456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12274328281088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 264983126 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚939864731054592 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12797148421632 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚834815770195456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 _runtime‚12745.349043258 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 209008496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚825724330926440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12273555636224 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119824677 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚847090098476544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 473991622 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚927067813298176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 790647808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 209008494 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 186284473 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚825723543238668 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 % _timestamp‚1.7885006457085848e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚847089307828736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚847090098476544 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 11129585664 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 186285029 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230665216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚813494190950224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚980 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119824675 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12273555636224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206725120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚813494173910864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 790647808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 473968604 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 264960110 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12229369327804 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚939864961719808 trainer/global_step‚11949 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 209008496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 264960110 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12796941696512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚554 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264983126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12274328281088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624ÔŒ÷æÅ4–./ _wandbruntime‚12746  _runtime‚12746'L£'—.‹N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119834325 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _step‚981 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12231141116302 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12230370467682 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 264981821 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 209025368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 474030233 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209025368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚939941349876224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186299357 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12274560381952 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 119834323 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚847158803174912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚834883470134784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚813560240346624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 264981821 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12798188235264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12275333040128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12274560381952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66465034 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚927143161640960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12797989912064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚847158012513792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 209025370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 474007189 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚813560257385984 $ _timestamp‚1.788500646603385e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 474007189 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186299902 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚12746.244224327 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚825791398502286 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 265004863 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 474030233 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚825790610814306 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 trainer/global_step‚11949 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 265004863 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚847158803174912 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66465577 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚927143137700864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209025370 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12274560381952 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 119834323 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚847158803174912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚834883470134784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚813560240346624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 772658176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 264981821 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12798188235264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12275333040128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12274560381952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66465034 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_ba5R©õ tch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚927143161640960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12797989912064 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚847158012513792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 209025370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 474007189 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚813560257385984 $ _timestamp‚1.788500646603385e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 474007189 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186299902 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚12746.244224327 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚825791398502286 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 265004863 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222263296 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 474030233 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚825790610814306 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 trainer/global_step‚11949 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 265004863 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚847158803174912 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66465577 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚927143137700864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209025370 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 _step‚981 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12231141116302 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12230370467682 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 264981821 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 209025368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 375175680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 474030233 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209025368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚939941349876224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 12155092992 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186299357 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543Õk¤94™./ _wandbruntime‚12746  _runtime‚12746ZTU^š.Ø 1loss/SRqqt2/detection_loss‚2.1866146120531806 0loss/SRqqt3/detection_loss‚2.094979896582663 2loss/SRqqt1/assignment_loss‚2.0013056349319718 /loss/FRt1/detection_loss‚2.2710986584424973 0loss/SRqqt1/detection_loss‚2.349391287813584 epoch‚9 trainer/global_step‚11949 .loss/FRt2/detection_loss‚2.575237516445372 0loss/FRt1/assignment_loss‚3.0315836388237623 1loss/FBt1/assignment_loss‚0.05731686677974371 .loss/FRt3/detection_loss‚2.865437907576561 .loss/FBt4/detection_loss‚0.429369022811662 &loss/total_loss‚38.915111366318634 2loss/SRqqt4/assignment_loss‚0.8272438692134607 1loss/FBt2/assignment_loss‚0.07201029561230143 .loss/FBt3/detection_loss‚2.404997306210654 _step‚982 2loss/SRqqt2/assignment_loss‚1.4013185308411205 1loss/FBt3/assignment_loss‚0.10670138046251461 1loss/FBt4/assignment_loss‚0.07120573243536268 1loss/SRqqt4/detection_loss‚0.7230876920562561 0loss/FRt4/assignment_loss‚0.8716802196959926 -loss/FBt2/detection_loss‚2.23832368850708 % _timestamp‚1.7885006466052856e+09 _runtime‚12746.245891159 2loss/SRqqt3/assignment_loss‚1.2964925430715084 /loss/FRt4/detection_loss‚1.4100922868421224 /loss/FBt1/detection_loss‚2.4570312182108562 0loss/FRt2/assignment_loss‚2.9097204623800335 0loss/FRt3/assignment_loss‚2.2628710985183718¿õ[Fc›.Ý 2loss/SRqqt3/assignment_loss‚1.2964925430715084 /loss/FRt4/detection_loss‚1.4100922868421224 /loss/FBt1/detection_loss‚2.4570312182108562 0loss/FRt2/assignment_loss‚2.9097204623800335 0loss/FRt3/assignment_loss‚2.2628710985183718 1loss/SRqqt2/detection_loss‚2.1866146120531806 0loss/SRqqt3/detection_loss‚2.094979896582663 2loss/SRqqt1/assignment_loss‚2.0013056349319718 /loss/FRt1/detection_loss‚2.2710986584424973 0loss/SRqqt1/detection_loss‚2.349391287813584 epoch‚9 trainer/global_step‚11949 .loss/FRt2/detection_loss‚2.575237516445372 0loss/FRt1/assignment_loss‚3.0315836388237623 1loss/FBt1/assignment_loss‚0.05731686677974371 .loss/FRt3/detection_loss‚2.865437907576561 .loss/FBt4/detection_loss‚0.429369022811662 &loss/total_loss‚38.915111366318634 2loss/SRqqt4/assignment_loss‚0.8272438692134607 1loss/FBt2/assignment_loss‚0.07201029561230143 .loss/FBt3/detection_loss‚2.404997306210654 _step‚982 2loss/SRqqt2/assignment_loss‚1.4013185308411205 1loss/FBt3/assignment_loss‚0.10670138046251461 1loss/FBt4/assignment_loss‚0.07120573243536268 1loss/SRqqt4/detection_loss‚0.7230876920562561 0loss/FRt4/assignment_loss‚0.8716802196959926 -loss/FBt2/detection_loss‚2.23832368850708 % _timestamp‚1.7885006466052856e+09 _runtime‚12746.245891159Ö:Ÿàœ.‚(:’ ªµéԷ gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.938' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.187599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™$‚.‚(:ô ªµéÔ츧°! disk.nvme0n1.in‚ 37814.0078125# disk.dm-0.out‚300389.6689453125 disk.sda3.out‚ 713.1015625 disk.dm-1.out‚ 713.1015625 network.sent‚ 3.0710301e+07 network.recv‚ 4.222222e+06) disk./.usagePercent‚78.76155326160972 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.7890625" proc.memory.rssMB‚ 2940.4765625$ disk./.usageGB‚1173.604061126709& disk.nvme0n1.out‚300389.6689453125 disk.dm-0.in‚ 37814.0078125) proc.memory.availableMB‚ 1121.62890625) proc.memory.percent‚17.94724464416504 cpu‚25.307516634550574 proc.cpu.threads‚17 disk.sda.out‚ 713.1015625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0$ memory_percent‚93.15412044525146 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 37814.0078125( disk.nvme0c0n1.out‚300389.6689453125ªBC|’ž.‚(:‡ ¹µéÔº¬²­ gpu.0.gpu‚24.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.819' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5638 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Öa3¤~Ÿ.‚(:ó ¹µéÔ¿ƒ£¯ disk.drbd1260.out‚ 1.7890625 disk.sda.out‚ 713.5703125 disk.sda1.in‚0! disk.nvme0n1.in‚ 37814.0078125 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚93.15285682678223" proc.memory.rssMB‚ 2940.4765625 cpu‚25.307259083044528 proc.cpu.threads‚17# disk.dm-0.out‚300880.5205078125 disk.sda3.out‚ 713.5703125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0( disk./.usagePercent‚78.7740049560601# disk.nvme0c0n1.in‚ 37814.0078125( disk.nvme0c0n1.out‚300880.5205078125& disk.nvme0n1.out‚300880.5205078125 disk.sda3.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 1121.8359375) proc.memory.percent‚17.94724464416504 network.sent‚ 3.0819537e+07 network.recv‚ 4.229107e+06% disk./.usageGB‚1173.7896003723145 disk.dm-0.in‚ 37814.0078125 disk.dm-1.out‚ 713.5703125?Ô0’ .‚(:‡ ȵéÔÀò€¯ gpu.0.gpu‚37.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹¾]ƒ¡.‚(:ø ȵéÔù„Ÿ¯ proc.cpu.threads‚17% disk./.usageGB‚1174.2896766662598 disk.sda3.out‚ 713.9921875 disk.dm-1.in‚0 disk.dm-1.out‚ 713.9921875 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚301232.22216796875 disk.sda3.in‚0 disk.drbd1260.out‚ 1.79296875 disk.sda1.in‚0! disk.nvme0n1.in‚ 37814.0078125' disk.nvme0n1.out‚301232.22216796875$ disk.dm-0.out‚301232.22216796875$ memory_percent‚93.15435886383057" proc.memory.rssMB‚ 2940.4765625 network.sent‚ 3.0827391e+07 network.recv‚ 4.233727e+06 disk.sda.in‚0 disk.sda.out‚ 713.9921875 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 37814.0078125 disk.dm-0.in‚ 37814.0078125) proc.memory.availableMB‚ 1121.58984375) disk./.usagePercent‚78.80756549573871) proc.memory.percent‚17.94724464416504 cpu‚25.306998007763729‹Õ4¢./ _wandbruntime‚12791  _runtime‚12791ò¬£éÉ£.à & lr-AdamW/pg1‚1.367155319212711e-07 & lr-AdamW/pg2‚1.367155319212711e-07 trainer/global_step‚11999 % _timestamp‚1.7885006915528724e+09 _runtime‚12791.193409412 _step‚983W`*Τ.È & lr-AdamW/pg1‚1.367155319212711e-07 & lr-AdamW/pg2‚1.367155319212711e-07 trainer/global_step‚11999 % _timestamp‚1.7885006915528724e+09 _runtime‚12791.193409412 _step‚983×[/Î4¥./ _wandbruntime‚12791  _runtime‚12791«èªù(¦.‚P ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚838200767161856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚850524556495360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 266049012 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚943684495080448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 266072028 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚816797495698864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚850524556495360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚930835230437376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚829076935611776 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12323807336448 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11999 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 209852196 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204627968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚838200767161856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187032751 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885006915541694e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚838200749158912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚838200749158912 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚943684723648512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 209852194 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚850525347143168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 209852194 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 266072028 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚829077723299548 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12324579981312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12323807336448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66726212 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120307077 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66725676 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 475924224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 187033289 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12279439912912 _runtime‚12791.194631344STƒK§.‡P G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚516 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 187033289 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12279439912912 _runtime‚12791.194631344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚838200767161856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚850524556495360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 266049012 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚943684495080448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 10558147584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 266072028 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3125 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚816797495698864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚489 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 10929229312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚23016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚850524556495360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚930835230437376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚829076935611776 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12323807336448 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11999 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1019674624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 209852196 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204627968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚838200767161856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187032751 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 772644864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885006915541694e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1005 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 12155092992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚838200749158912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 QIDeviceStatsMonitor.on_train_batch_start/inactive_splië` t.small_pool.current‚536 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚23018 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚838200749158912 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚943684723648512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 770648412 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 209852194 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚850525347143168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 772644864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 209852194 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 266072028 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228568064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 375175680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚829077723299548 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12324579981312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12323807336448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66726212 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120307077 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66725676 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 475924224ØÚ:]Ÿ4¨./ _wandbruntime‚12792  _runtime‚12792µÞä'©.ŒN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12324812082176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12281211701410 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209869068 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 266093765 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 266070723 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚838268467101184 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12325584740352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66730965 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚930910578780160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 475962835 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚838268449098240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚943761112359936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 475962835 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 trainer/global_step‚11999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚838268449098240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚850594051841536 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 266070723 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12850533579776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 475939791 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209869070 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187047688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚850593261180416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120316723 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12280441052790 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 475939791 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚930910554840064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12325584740352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12324812082176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 % _timestamp‚1.7885006924114962e+09Àï'ª.‘N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 209869070 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 10558147584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚23042 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 266070723 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚838268467101184 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12325584740352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66730965 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 770648620 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 177240576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚930910578780160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 10929229312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 475962835 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1046 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚838268449098240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚943761112359936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 _step‚985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202517504 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 11129585664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 772658176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 475962835 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 790661120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚27846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 12155092992 trainer/global_step‚11999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚838268449098240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1005 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1005 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚850594051841536 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 266070723 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12850533579776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 12155092992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1025507328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 475939791 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209869070 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187047688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚850593261180416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚23042 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120316723 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12280441052790 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1017634825 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 475939791 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 10929229312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚930910554840064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 11892004352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12325584740352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12324812082176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 312660992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3125 % _timestamp‚1.7885006924114962e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12324812082176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12281211701410 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209869068 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 266093765 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚853959805161984 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚820100817486864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚820100834526224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 210695894 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120789475 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12374059036672 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚947504376856064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚12838.020648778 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚934602623636480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12374831681536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187781020 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚947504139899392 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236956672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 267160930 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚853960595809792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚853960595809792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚841585764128256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12901516262912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 213016576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚12049 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 210695896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66991545 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 267137914 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 477833808 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 267137914 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12329510498020 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 210695896Íñæ6 (¹.‡P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 11892004352 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1046 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236956672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 11892004352 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 11129585664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 267160930 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚853960595809792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚853960595809792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚841585764128256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12901516262912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 213016576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚28115 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚12049 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3125 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚23018 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚28115 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 10929229312 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 210695896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66991545 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1005 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚23016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 267137914 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 787687772 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 11518882825 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 477833808 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 267137914 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12329510498020 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1017634825 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 210695896 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚853959805161984 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚820100817486864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚820100834526224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚27846 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 210695894 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120789475 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12374059036672 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1075 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚947504376856064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1005 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚516 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚489 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚12838.020648778 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚934602623636480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 12155092992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12374831681536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚27846 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187781020 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚39 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚947504139899392 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 12155092992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1005Üì æô4º./ _wandbruntime‚12838  _runtime‚12838á%®B'».ŒN XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 210712770 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 267159625 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 477895437 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 477895437 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120799125 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚947580756037120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚832497395560522 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12330511637898 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12375836440576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 267182667 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187796421 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 210712770 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12375836440576 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12375063782400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66997296 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187795894 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210712768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12902760117760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 210712768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚854029300508160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 477872393 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚947580533773824 % _timestamp‚1.7885007392557502e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚841653446064640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚841653464067584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 477872393 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66996771 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚820166900961984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚820166883922624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚854029300508160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚12049 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12375063782400 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _step‚989êõé²¼.‘N ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1005 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 11892004352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12375063782400 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 790661120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚489 _step‚989 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1025507328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 210712770 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 267159625 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚39 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 477vF]ee895437 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 11518882825 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1019674624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1019674624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 477895437 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 11129585664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚516 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120799125 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚947580756037120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚832497395560522 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12330511637898 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚489 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12375836440576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 787687980 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 267182667 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚28115 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187796421 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1075 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚28115 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 210712770 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12375836440576 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12375063782400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66997296 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187795894 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚23044 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210712768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12902760117760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚516 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 210712768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚854029300508160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 477872393 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚947580533773824 % _timestamp‚1.7885007392557502e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚841653446064640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚841653464067584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 477872393 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66996771 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚820166900961984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚820166883922624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚854029300508160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚489 trainer/global_step‚12049 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 198323200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1005ÝÖpÍï4½./ _wandbruntime‚12838  _runtime‚12838«³èF\¾.Ö /loss/FRt2/detection_loss‚2.3149693398682367 /loss/FBt2/detection_loss‚2.1733214530077847 0loss/FBt4/detection_loss‚0.47892032814128566 %loss/total_loss‚39.64881534426464 1loss/SRqqt3/assignment_loss‚1.470703314814084 1loss/FBt1/assignment_loss‚0.20255720078874498 -loss/FRt1/detection_loss‚2.60243066473815 /loss/FBt3/detection_loss‚2.2300609312951565 trainer/global_step‚12049 /loss/FBt1/detection_loss‚2.8300635864337287 /loss/FRt2/assignment_loss‚3.424793059485299 1loss/SRqqt1/assignment_loss‚1.161957774417741 /loss/FBt2/assignment_loss‚0.149686314350447 1loss/SRqqt4/detection_loss‚0.7564522543886922 _runtime‚12838.898480737 /loss/FRt1/assignment_loss‚3.095285243109653 2loss/SRqqt4/assignment_loss‚0.8269380684854146 0loss/FBt3/assignment_loss‚0.3816020293497209 _step‚990 0loss/FRt4/assignment_loss‚1.0194260105490685 1loss/SRqqt3/detection_loss‚1.7606043195668257 epoch‚9 % _timestamp‚1.7885007392583072e+09 /loss/FRt3/assignment_loss‚2.864135214260646 /loss/FRt3/detection_loss‚2.8405983628971234 1loss/SRqqt1/detection_loss‚2.0473191261291506 1loss/SRqqt2/detection_loss‚2.2748558305203916 2loss/SRqqt2/assignment_loss‚1.2796662862823722 1loss/FBt4/assignment_loss‚0.07108361798102199 /loss/FRt4/detection_loss‚1.3913850134039032û,Na¿.Û 1loss/SRqqt3/assignment_loss‚1.470703314814084 1loss/FBt1/assignment_loss‚0.20255720078874498 -loss/FRt1/detection_loss‚2.60243066473815 /loss/FBt3/detection_loss‚2.2300609312951565 trainer/global_step‚12049 /loss/FBt1/detection_loss‚2.8300635864337287 /loss/FRt2/assignment_loss‚3.424793059485299 1loss/SRqqt1/assignment_loss‚1.161957774417741 /loss/FBt2/assignment_loss‚0.149686314350447 1loss/SRqqt4/detection_loss‚0.7564522543886922 _runtime‚12838.898480737 /loss/FRt1/assignment_loss‚3.095285243109653 2loss/SRqqt4/assignment_loss‚0.8269380684854146 0loss/FBt3/assignment_loss‚0.3816020293497209 _step‚990 0loss/FRt4/assignment_loss‚1.0194260105490685 1loss/SRqqt3/detection_loss‚1.7606043195668257 epoch‚9 % _timestamp‚1.7885007392583072e+09 /loss/FRt3/assignment_loss‚2.864135214260646 /loss/FRt3/detection_loss‚2.8405983628971234 1loss/SRqqt1/detection_loss‚2.0473191261291506 1loss/SRqqt2/detection_loss‚2.2748558305203916 2loss/SRqqt2/assignment_loss‚1.2796662862823722 1loss/FBt4/assignment_loss‚0.07108361798102199 /loss/FRt4/detection_loss‚1.3913850134039032 /loss/FRt2/detection_loss‚2.3149693398682367 /loss/FBt2/detection_loss‚2.1733214530077847 0loss/FBt4/detection_loss‚0.47892032814128566 %loss/total_loss‚39.64881534426464Þê³’À.‚(:‡ „¶éÔÉп¯ gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.157' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2314 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Hë ‡Á.‚(:ü „¶éÔ‹À´±) disk./.usagePercent‚78.80756216764078% disk./.usageGB‚1174.2896270751953 disk.sda1.in‚0 disk.dm-0.in‚37951.92578125 disk.sda3.in‚0) proc.memory.percent‚17.94724464416504 proc.cpu.threads‚17 disk.sda1.out‚0$ disk.nvme0c0n1.in‚37951.92578125( disk.nvme0c0n1.out‚302518.3212890625" disk.nvme0n1.in‚37951.92578125" proc.memory.rssMB‚ 2940.4765625 cpu‚25.305970065380123 disk.sda.in‚0 disk.sda.out‚ 716.08984375 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.80078125) proc.memory.availableMB‚ 1121.27734375 network.sent‚ 3.0951089e+07 disk.sda2.out‚0& disk.nvme0n1.out‚302518.3212890625# disk.dm-0.out‚302518.3212890625 disk.sda3.out‚ 716.08984375 disk.dm-1.out‚ 716.08984375 disk.drbd1260.in‚0$ memory_percent‚93.15626621246338 network.recv‚ 4.253669e+06°0ø’Â.‚(:‡ “¶éÔÙ‰Ž° gpu.0.gpu‚42.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.683' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.7366 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(³ÁñwÃ.‚(:ì “¶éÔªèѱ# disk.dm-0.out‚302869.7119140625 network.sent‚ 3.1061117e+07 disk.sda.out‚ 716.484375 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 38084.953125 disk.dm-0.in‚ 38084.953125# memory_percent‚93.1553840637207' proc.memory.availableMB‚ 1121.421875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0" disk.nvme0c0n1.in‚ 38084.953125( disk.nvme0c0n1.out‚302869.7119140625 proc.cpu.threads‚17% disk./.usageGB‚1174.2895889282227 disk.sda3.out‚ 716.484375 disk.drbd1260.out‚ 1.80078125" proc.memory.rssMB‚ 2940.4765625) proc.memory.percent‚17.94724464416504 cpu‚25.30574945414523 network.recv‚ 4.260034e+06) disk./.usagePercent‚78.80755960756544 disk.sda3.in‚0 disk.dm-1.out‚ 716.484375 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚302869.7119140625sÉ«Ä.‚(:’ ¢¶éÔÞ‰° gpu.0.gpu‚49.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.397000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sŸ£[Å.‚(:Ð ¢¶éÔçü­² disk.sda3.in‚0 disk.sda1.in‚0 disk.nvme0n1.out‚303151) proc.memory.percent‚17.94724464416504 proc.cpu.threads‚17) disk./.usagePercent‚78.79616317617119 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.out‚303151 cpu‚25.305508357289806 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0c0n1.out‚303151 disk.nvme0n1.in‚ 38084.953125 disk.dm-0.in‚ 38084.953125) proc.memory.availableMB‚ 1121.37109375" proc.memory.rssMB‚ 2940.4765625 network.recv‚ 4.263657e+06 disk.sda3.out‚ 716.8515625 disk.dm-1.out‚ 716.8515625 disk.drbd1260.out‚ 1.8046875 disk.sda.in‚0 disk.sda.out‚ 716.8515625" disk.nvme0c0n1.in‚ 38084.953125$ memory_percent‚93.15569400787354 network.sent‚ 3.1064272e+07$ disk./.usageGB‚1174.119773864746é „Æ.‚(:’ ±¶éÔòùã± gpu.0.gpu‚57.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.938' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.187599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ^qÇ.‚(:æ ±¶éÔ©’ð³ disk.sda3.in‚0$ memory_percent‚93.15719604492188$ proc.memory.availableMB‚1121.125 cpu‚25.307732287202736 proc.cpu.threads‚17 network.recv‚ 4.268151e+06) disk./.usagePercent‚78.70824302070211 disk.sda2.in‚0% disk.nvme0c0n1.out‚303310.2265625 disk.sda3.out‚ 717.23828125 disk.dm-1.out‚ 717.23828125 disk.drbd1260.out‚ 1.8046875 disk.sda.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 38084.953125# disk.nvme0n1.out‚303310.2265625 disk.drbd1260.in‚0) proc.memory.percent‚17.94724464416504 disk.dm-0.in‚ 38084.953125 network.sent‚ 3.1070576e+07 disk.sda1.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 38084.953125 disk.dm-1.in‚0 disk.sda.out‚ 717.23828125" proc.memory.rssMB‚ 2940.4765625$ disk./.usageGB‚1172.809700012207 disk.dm-0.out‚303310.2265625:u·Ï‘È.‚(:† À¶éÔè¢Ä® gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.281' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0562 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sM|É.‚(:ñ À¶éÔð ß± disk.dm-0.in‚ 38084.953125" disk.dm-0.out‚303948.771484375 proc.cpu.threads‚17% disk./.usageGB‚1173.1904907226562 disk.drbd1260.out‚ 1.80859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 38084.953125# memory_percent‚93.1572437286377 disk.sda3.in‚0 disk.dm-1.out‚ 717.74609375 disk.sda.out‚ 717.74609375' disk.nvme0c0n1.out‚303948.771484375( proc.memory.availableMB‚ 1121.1171875" proc.memory.rssMB‚ 2940.4765625) proc.memory.percent‚17.94724464416504 cpu‚25.313279060041047 network.sent‚ 3.1078409e+07) disk./.usagePercent‚78.73379820478506 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 38084.953125 network.recv‚ 4.272594e+06 disk.sda3.out‚ 717.74609375 disk.dm-1.in‚0 disk.sda.in‚0% disk.nvme0n1.out‚303948.771484375AR‘Ê.‚(:† ϶éÔ®ûÕ­ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚33.0 gpu.0.powerWatts‚114.882' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9764 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ8`¾zË.‚(:ï ϶éÔ¤ù’° network.recv‚ 4.275538e+06( disk./.usagePercent‚78.7337984607926 disk.drbd1260.out‚ 1.80859375' disk.nvme0c0n1.out‚304205.107421875 cpu‚25.31867600154617% disk./.usageGB‚1173.1904945373535" disk.dm-0.out‚304205.107421875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚38235.69921875% disk.nvme0n1.out‚304205.107421875( proc.memory.availableMB‚ 1121.1171875 network.sent‚ 3.1082597e+07 disk.dm-0.in‚38235.69921875 disk.sda3.in‚0 disk.sda3.out‚ 718.015625 disk.drbd1260.in‚0 disk.sda.out‚ 718.015625 disk.sda1.out‚0$ disk.nvme0c0n1.in‚38235.69921875 disk.dm-1.in‚0 disk.dm-1.out‚ 718.015625 disk.sda.in‚0# memory_percent‚93.1572437286377" proc.memory.rssMB‚ 2940.4765625) proc.memory.percent‚17.94724464416504 proc.cpu.threads‚17Q™É ‘Ì.‚(:† Þ¶éÔ¦óý® gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚14.79499816894531* gpu.0.memoryAllocatedBytes‚ 12708806656 gpu.0.temp‚33.0 gpu.0.powerWatts‚115.129' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0258 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉŽ´xÍ.‚(:í Þ¶éԉЀ°' proc.memory.availableMB‚ 1121.359375) proc.memory.percent‚17.94724464416504 proc.cpu.threads‚17% disk./.usageGB‚1173.1904945373535 disk.drbd1260.out‚1.8125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0n1.in‚38235.69921875$ disk.nvme0n1.out‚304582.45703125 disk.dm-0.in‚38235.69921875 network.sent‚ 3.108761e+07 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚38235.69921875! disk.dm-0.out‚304582.45703125$ memory_percent‚93.15576553344727" proc.memory.rssMB‚ 2940.4765625( disk./.usagePercent‚78.7337984607926 disk.sda3.out‚ 718.49609375 disk.dm-1.out‚ 718.49609375 disk.drbd1260.in‚0 cpu‚25.32422761977469 network.recv‚ 4.279159e+06 disk.dm-1.in‚0 disk.sda.out‚ 718.49609375& disk.nvme0c0n1.out‚304582.45703125Ÿu(E4Î./ _wandbruntime‚12937  _runtime‚12937@D==o*Ï.éT \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 11892004352 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚516 trainer/global_step‚12089 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 212023356 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18215 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 375175680 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 268458831 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚952961610881536 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚12438822478848 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 121576063 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚939993858749440 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚186 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 268440616 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚859027618034688 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚12438242434560 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28115 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 212023356 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882825 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 480482189 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 212023358 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚67458315 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚824882054557088 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚12393469133070 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 580044288 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 126695936 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚837275506650798 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 12155092992 _step‚991 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3125 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚837276101762662 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚12394047205574 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 10929229312 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚12967752132096 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚846589393603072 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27846 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12155092992 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12155092992 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 268458831 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 189034564 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚846589375600128 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578072504 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1005 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 480463972 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18215 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 268440616 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚184 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 _runtime‚12937.983139477 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚12967625436160 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚12438822478848 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1005 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18217 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 % _timestamp‚1.7885008383424122e+09 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚859028216081920 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 598047232 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚846589393603072 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚939993834809344 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1005 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 11892004352 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 580044288 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111864 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚824882037517728 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634825 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚846589375600128 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚12438242434560 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 480482189 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 480463972 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚859027618034688 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0ø¨ý+Ð.îT eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚824882054557088 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚12393469133070 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1025507328 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 580044288 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 126695936 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚837275506650798 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 12155092992 _step‚991 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3125 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚837276101762662 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚12394047205574 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚489 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 10929229312 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚12967752132096 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚846589393603072 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚27846 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚39 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 12155092992 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 12155092992 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 11129585664 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 268458831 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 189034564 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚846589375600128 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1019674624 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 578072504 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 11129585664 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1005 K=DeviceStatsMonitÉlÑIor.on_validation_batch_start/active.all.freed‚ 480463972 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚18215 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 268440616 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚184 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚489 _runtime‚12937.983139477 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3125 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚12967625436160 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1005 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚12438822478848 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1005 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚18217 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 177240576 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚516 % _timestamp‚1.7885008383424122e+09 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚859028216081920 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 598047232 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚846589393603072 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚939993834809344 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1005 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 11892004352 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 580044288 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 595111864 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚824882037517728 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1017634825 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1025507328 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚516 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚846589375600128 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚12438242434560 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 480482189 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 480463972 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚859027618034688 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1025507328 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1075 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 11892004352 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚516 trainer/global_step‚12089 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 212023356 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚18215 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 375175680 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 268458831 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1046 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚952961610881536 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚489 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚12438822478848 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 121576063 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚939993858749440 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 312660992 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚186 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 268440616 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚859027618034688 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚12438242434560 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚28115 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 212023356 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 11518882825 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 480482189 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1019674624 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 212023358 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚67458315ß4˜4Ñ./ _wandbruntime‚12939  _runtime‚12939´¼D£y)Ò.óR ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3125 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 11892004352 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 580044288 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 121584043 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1046 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚516 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 212036330 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1019674624 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚27846 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚12394510366538 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚12968090184704 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1025507328 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚859073550782464 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 268467125 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 150636032 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 126695936 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚824924350397184 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 12155092992 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚516 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 598047232 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 580044288 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚28115 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚39 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1005 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚12438707183104 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 189047702 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1075 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚12438707183104 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚184 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18215 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 268448910 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1005 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚824924333357824 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 480503455 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 10929229312 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚28115 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚12438707183104 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚184 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 11129585664 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚18215 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 268448910 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1005 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚824924333357824 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 480503455 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 10929229312 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1005 B&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.004051620606333017 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26213690638542175 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6369046568870544 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5091401934623718 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0021919880528002977 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42258068919181824 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4059435725212097 9"Purity/2Faÿ¡ùRt*SRqqt1FBt/SRqqt_purity‚0.6345464587211609 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3486606776714325 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_4_of_4‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.267815500497818 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 jet/accuracy_6_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5531609058380127 )jet/accuracy_1_of_5‚0.968465268611908 particle/accuracy_6_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6569443941116333 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.585286021232605 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41027021408081055 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31821438670158386 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2269689291715622 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5455830097198486 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6578947305679321 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4603184163570404 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.041079625487327576 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 particle/accuracy_4_of_7‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_1_of_7‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_3_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876126646995544 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.138888880610466 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5231638550758362 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1810699701309204 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.13461539149284363 +jet/accuracy_2_of_5‚0.47737452387809753 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03558911010622978 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.039086632430553436 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5396113395690918 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.47472772002220154 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_6‚1 jet/accuracy_5_of_7‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5728686451911926 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6014636158943176 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9898893237113953 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.08625730872154236 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_5_of_5‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.010080644860863686 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 _step‚993 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5430556535720825 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ,particle/sensitivity‚0.34381550550460815 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9919828176498413 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2520114779472351 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5133333206176758 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14763659238815308 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 jet/accuracy_5_of_6‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.016125062480568886 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.45075759291648865 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2444184273481369 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892492890357971 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.2244444489479065 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3186092972755432 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5324267745018005 :"Purity/*FRt3SRqqt0FBt/event_purity‚0.01036866381764412 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12144876271486282 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968628287315369 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39537036418914795 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 +jet/accuracy_3_of_5‚0.08701995760202408 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6617063879966736 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.22727273404598236 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2147587537765503 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.021756110712885857 -jet/accuracy_4_of_4‚0.0073660085909068584 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4814814329147339 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18881121277809143 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5528155565261841 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5447530746459961 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2985104024410248 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.48480114340782166 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5186967253684998 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14251071214675903 ,"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19956503808498383 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2316763550043106 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5321173667907715 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009542882442474365 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34583333134651184 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2232142835855484 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_6_of_6‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5479972958564758 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27853867411613464 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09181547164916992 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2759994864463806 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5214591026306152 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19238342344760895 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6033533811569214 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026462005451321602 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17171718180179596 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24540840089321136 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.038063112646341324 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5779762864112854 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.037484828382730484 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347517371177673 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.990687906742096 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5048863291740417 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 :"Purity/0FRt3SRqqt0FBt/event_purity‚0.03267974033951759 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11046439409255981 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3871252238750458 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.366940975189209 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9944444298744202 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5459656119346619 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5838202834129333 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10218576341867447 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12769059836864471 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893950819969177 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.028431378304958344 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49748721718788147 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20805931091308594 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24198855459690094 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5394842624664307 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7658808827400208 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5368074178695679 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9929705262184143 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5415229201316833 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/event_purity‚0.03267974033951759 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12209483981132507 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2632371485233307 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4141404330730438 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2647324204444885 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5816692113876343 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2455998659133911 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0031746032182127237 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.317592591047287 ,"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1264936923980713 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4849475920200348 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_2_of_4‚0.3903317153453827 particle/accuracy_3_of_3‚1 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19277483224868774 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2577160894870758 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.20028452575206757 ,jet/accuracy_3_of_3‚0.029731545597314835 ( Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9859127402305603 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.020011337473988533 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9960488080978394 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4471890330314636 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 _runtime‚12953.313131445 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15476840734481812 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2068965882062912 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8162113428115845 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.45679011940956116 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1800515502691269 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.44054052233695984 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26238343119621277 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8918128609657288 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5197597742080688 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.10775861889123917 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333730697632 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5608971118927002 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 'particle/f_score‚0.4219611883163452 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.921539843082428 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.709140956401825 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.505363941192627 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5098522305488586 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9820341467857361 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.49917837977409363 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 )jet/accuracy_1_of_1‚0.494660884141922 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47766783833503723 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974868893623352 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5628982186317444 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2269689291715622 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9919427633285522 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5455814599990845 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5327492952346802 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 trainer/global_step‚12089 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 +jet/accuracy_2_of_2‚0.11913243681192398 particle/accuracy_5_of_6‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10599356144666672 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 $ _timestamp‚1.788500853670861e+09 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.991504430770874 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4594275653362274 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5030010342597961 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3436637818813324 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5149646401405334 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4979531466960907 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6461235284805298 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5053150057792664 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22258426249027252 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.42604169249534607 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5601851940155029 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.5448717474937439 jet/accuracy_2_of_7‚0.5 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.914893627166748 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8530091047286987 jet/accuracy_7_of_7‚0 particle/accuracy_5_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9818594455718994 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5540722012519836 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5354549288749695 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3117816150188446 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1804848313331604 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9472426772117615 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.49249255657196045 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27084416151046753 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04308176040649414 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2534421682357788 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15096312761306763 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09376340359449387 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36218103766441345 particle/accuracy_2_of_6‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08289888501167297 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1564861536026001 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5604903697967529 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.020762929692864418 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2227945178747177 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07789409905672073 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.929894208908081 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.49917837977409363 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5641292929649353 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_6‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31843021512031555 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9879351258277893 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5580635666847229 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4547130763530731 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12485827505588531 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_5‚1 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBs^¾4t/SRqqt_purity‚0.5744677186012268 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1998257040977478 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9903280138969421 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 )loss/val_total_loss‚39.36630463195116 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 *jet/accuracy_1_of_3‚0.8890257477760315 +jet/accuracy_3_of_4‚0.08012168109416962 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984179139137268 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5390260815620422 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816w«b°byØ.§ @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5838202834129333 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10218576341867447 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12769059836864471 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893950819969177 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.028431378304958344 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49748721718788147 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20805931091308594 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24198855459690094 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5394842624664307 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7658808827400208 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5368074178695679 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9929705262184143 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5415229201316833 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/event_purity‚0.03267974033951759 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12209483981132507 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2632371485233307 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4141404330730438 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2647324204444885 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5816692113876343 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2455998659133911 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0031746032182127237 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.317592591047287 ,"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1264936923980713 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4849475920200348 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_2_of_4‚0.3903317153453827 particle/accuracy_3_of_3‚1 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19277483224868774 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2577160894870758 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.20028452575206757 ,jet/accuracy_3_of_3‚0.029731545597314835 ( Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9859127402305603 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.020011337473988533 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9960488080978394 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4471890330314636 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 _runtime‚12953.313131445 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15476840734481812 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2068965882062912 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8162113428115845 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.45679011940956116 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1800515502691269 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.44054052233695984 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26238343119621277 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8918128609657288 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5197597742080688 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.10775861889123917 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333730697632 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5608971118927002 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 'particle/f_score‚0.4219611883163452 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.921539843082428 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.709140956401825 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.505363941192627 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5098522305488586 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9820341467857361 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.49917837977409363 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 )jet/accuracy_1_of_1‚0.494660884141922 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47766783833503723 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974868893623352 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5628982186317444 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2269689291715622 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9919427633285522 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5455814599990845 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5327492952346802 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 trainer/global_step‚12089 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 +jet/accuracy_2_of_2‚0.11913243681192398 particle/accuracy_5_of_6‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10599356144666672 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 $ _timestamp‚1.788500853670861e+09 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.991504430770874 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4594275653362274 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5030010342597961 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3436637818813324 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5149646401405334 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4979531466960907 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6461235284805298 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5053150057792664 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22258426249027252 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.42604169249534607 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5601851940155029 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.5448717474937439 jet/accuracy_2_of_7‚0.5 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.914893627166748 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8530091047286987 jet/accuracy_7_of_7‚0 particle/accuracy_5_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9818594455718994 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5540722012519836 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5354549288749695 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3117816150188446 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1804848313331604 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9472426772117615 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.49249255657196045 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27084416151046753 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04308176040649414 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2534421682357788 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15096312761306763 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09376340359449387 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36218103766441345 particle/accuracy_2_of_6‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08289888501167297 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1564861536026001 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5604903697967529 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.020762929692864418 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2227945178747177 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.07789409905672073 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.929894208908081 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.49917837977409363 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5641292929649353 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_6‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31843021512031555 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9879351258277893 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5580635666847229 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4547130763530731 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12485827505588531 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_5‚1 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5744677186012268 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1998257040977478 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9903280138969421 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 )loss/val_total_loss‚39.36630463195116 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 *jet/accuracy_1_of_3‚0.8890257477760315 +jet/accuracy_3_of_4‚0.08012168109416962 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984179139137268 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5390260815620422 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5701754689216614 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24579639732837677 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04217229783535004 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.006321839056909084 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5232558250427246 particle/accuracy_1_of_2‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36794400215148926 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12269555032253265 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17956143617630005 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17595110833644867 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5861155986785889 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9236111640930176 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9912051558494568 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911176562309265 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.639285683631897 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.47440090775489807 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 epoch‚9 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7121211886405945 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.004051620606333017 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26213690638542175 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6369046568870544 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5091401934623718 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0021919880528002977 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42258068919181824 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4059435725212097 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6345464587211609 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3486606776714325 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_4_of_4‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.267815500497818 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 jet/accuracy_6_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5531609058380127 )jet/accuracy_1_of_5‚0.968465268611908 particle/accuracy_6_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6569443941116333 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.585286021232605 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41027021408081055 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31821438670158386 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2269689291715622 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5455830097198486 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6578947305679321 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4603184163570404 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.041079625487327576 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 particle/accuracy_4_of_7‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_1_of_7‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_3_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876126646995544 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.138888880610466 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5231638550758362 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1810699701309204 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.13461539149284363 +jet/accuracy_2_of_5‚0.47737452387809753 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03558911010622978 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.039086632430553436 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5396113395690918 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.47472772002220154 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_6‚1 jet/accuracy_5_of_7‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5728686451911926 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6014636158943176 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9898893237113953 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.08625730872154236 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_5_of_5‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.010080644860863686 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 _step‚993 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5430556535720825 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ,particle/sensitivity‚0.34381550550460815 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9919828176498413 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2520114779472351 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5133333206176758 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14763659238815308 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 jet/accuracy_5_of_6‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.016125062480568886 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.45075759291648865 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2444184273481369 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892492890357971 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.2244444489479065 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3186092972755432 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5324267745018005 :"Purity/*FRt3SRqqt0FBt/event_purity‚0.01036866381764412 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12144876271486282 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968628287315369 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939104318618774 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.39537036418914795 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 +jet/accuracy_3_of_5‚0.08701995760202408 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6617063879966736 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.22727273404598236 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2147587537765503 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.021756110712885857 -jet/accuracy_4_of_4‚0.0073660085909068584 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4814814329147339 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18881121277809143 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5528155565261841 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5447530746459961 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2985104024410248 :"Purity/*FRt3SRq:Ô Bqt0FBt/SRqqt_purity‚0.48480114340782166 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5186967253684998 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14251071214675903 ,"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19956503808498383 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2316763550043106 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5321173667907715 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009542882442474365 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34583333134651184 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2232142835855484 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_6_of_6‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5479972958564758 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27853867411613464 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.09181547164916992 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2759994864463806 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5214591026306152 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19238342344760895 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6033533811569214 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026462005451321602 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17171718180179596 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24540840089321136 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.038063112646341324 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5779762864112854 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.037484828382730484 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347517371177673 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.990687906742096 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5048863291740417 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 :"Purity/0FRt3SRqqt0FBt/event_purity‚0.03267974033951759 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11046439409255981 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3871252238750458 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.366940975189209 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9944444298744202 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5459656119346619 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1á÷­ÈRÙ. wueegkhmj@ ú¶éÔÈÌé¸0`Trainer.fit` stopped: `max_epochs=10` reached. à‘¤¿‘Ú.‚(:† ü¶éÔóö›® gpu.0.gpu‚45.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚0.7373809814453125( gpu.0.memoryAllocatedBytes‚ 633405440 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.673' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v°˜H~Û.‚(:ó ü¶éÔ°êׯ network.recv‚ 4.290656e+06( disk./.usagePercent‚78.7360684795987% disk./.usageGB‚1173.2243194580078 disk.sda3.in‚0 disk.sda3.out‚ 719.1953125 disk.drbd1260.in‚0 disk.sda2.out‚0! disk.dm-0.out‚305516.16015625 disk.sda.out‚ 719.1953125 disk.sda1.in‚0" disk.nvme0n1.in‚38235.69921875) proc.memory.availableMB‚ 1212.56640625" proc.memory.rssMB‚ 3405.9453125 cpu‚25.332838661383597 proc.cpu.threads‚17 network.sent‚ 3.1200469e+07 disk.dm-1.in‚0 disk.sda2.in‚0$ disk.nvme0n1.out‚305516.16015625 disk.dm-0.in‚38235.69921875* proc.memory.percent‚20.788240432739258 disk.dm-1.out‚ 719.1953125 disk.drbd1260.out‚ 1.81640625 disk.sda.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚38235.69921875& disk.nvme0c0n1.out‚305516.16015625$ memory_percent‚92.59908199310303<›oŸÜ. wueegkhmjŒ ü¶éÔ¸ÊðzEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:20:07 • 0:00:00 1.01it/s v_num: gkhmå6Ï3%Ý. wueegkhmj ü¶éÔÀ”¬ð Kk›4Þ./ _wandbruntime‚12962  _runtime‚12962çm‚ß.‚(’¢e